Learning to Segment from Noisy Annotations: A Spatial Correction Approach
Authors: Jiachen Yao
Yikai Zhang
Songzhu Zheng
Mayank Goswami
Prateek Prasanna
Chao Chen
Venue: ICLR 2023 poster (2023)

Abstract: Noisy labels can significantly affect the performance of deep neural networks (DNNs). In medical image segmentation tasks, annotations are error-prone due to the high demand in annotation time and in the annotators' expertise. Existing methods mostly tackle label noise in classification tasks. Their independent-noise assumptions do not fit label noise in segmentation task. In this paper, we propose a novel noise model for segmentation problems that encodes spatial correlation and bias, which are prominent in segmentation annotations. Further, to mitigate such label noise, we propose a label correction method to recover true label progressively. We provide theoretical guarantees of the correctness of the proposed method. Experiments show that our approach outperforms current state-of-the-art methods on both synthetic and real-world noisy annotations.

Published as a conference paper at ICLR 2023
LEARNING TO SEGMENT FROM NOISY ANNOTATIONS: A SPATIAL CORRECTION APPROACH
Jiachen Yao', Yikai Zhang", Songzhu Zheng', Mayank Goswami", Prateek Prasanna', Chao Chen Stony Brook University, "Morgan Stanley, 'CUNY Queens College
I [jiachen. yao, zheng. songzhu, prateek. prasanna, chao.chen, 1) 1eeeen..eesttobboooe Jestonybrook.edu Pyikai. zhangemorganstanleey. com, mayank. is @gmail com
ABSTRACT
Noisy labels can significantly affect the performance of deep neural networks (DNNs). In medical image segmentation tasks, annotations are error-prone due to the high demand in annotation time and in the annotators' expertise. Exist- ing methods mostly assume noisy labels in different pixels are Li.d. However, segmentation label noise usually has strong spatial correlation and has prominent bias in distribution. In this paper, we propose a novel Markov model for segmen- tation noisy annotations that encodes both spatial correlation and bias. Further, to mitigate such label noise, we propose a label correction method to recover true label progressively. We provide theoretical guarantees of the correctness of the proposed method. Experiments show that our approach outperforms current state- of-the-art methods on both synthetic and real- world noisy annotations.
INTRODUCTION
Noisy annotations are inevitable large scale datasets, and can heavily impair the performance of deep neural networks (DNNs) due to their strong memorization power (Zhang et al.) [0166] Arpit (ct al. (0177). Image segmentation also suffers from the label noise problem. For medical images, segmentation quality is highly dependent on human annotators' expertise and time spent. In practice, medical students and residents in training are often recruited to annotate, potentially introducing er- rors (Gurari et al. 2013 Kohli et al., 2017). We also note even among experts, there can be poor consensus in terms of objects' location and boundary (Menze et al., 2014 Joskowicz et al.) 2018 Zhang et al. 2020a). Furthermore, segmentation annotations require pixel/loxel-level detailed de- lineations of the objects of interest. Annotating objects involving complex boundaries and structures are especially time-consuming. Thus, errors can naturally be introduced when annotating at scale.
Segmentation is the first step of most analysis pipelines. Inaccurate segmentation can introduce error into measurements such the morphology, which can be important for downstream diagnosis and prognostic tasks (Wang et al., 2019a Nafe et al.) 2005). Therefore, it is important to develop robust training methods against segmentation label noise. However, despite many existing methods addressing label noise in classification tasks (Patrini et al. 2017 Yu al. 2019: Zhang & Sabuncu, 2018 Li et al. 2020) Liu et al.) 2020 Zhang et al. 2021 Xia et al. 2021) limited progress has been made in the context of image segmentation.
A few existing segmentation label noise approaches (Zhu et al.] 2019 Zhang et al. 2020b a directly apply methods in classification label noise. However, these methods assume the label noise for each pixel is i.i.d. (independent and identically distributed). This assumption is not realistic in the seg- mentation context, where annotation is often done by brushes, and error is usually introduced near the boundary of objects. Regions further away from the boundary are less likely to be mislabeled (see Fig. Tc for illustration). Therefore, in segmentation tasks, label noise of pixels has to be spatially correlated. An i.i.d. label noise will result in unrealistic annotations as in Fig. Tb
We propose novel label noise model for segmentation annotations. Our model simulates the real annotation scenario, where an annotator uses a brush to delineate the boundary of an object. The noisy boundary can be considered a random yet continuous distortion of the true boundary. To
Codes available at htt.ppee/ggthlecccco/maaeboobb/a

---

Published as a conference paper at ICLR 2023
capture this noise behavior, we propose Markov process model. At each step of the process, two Bernoulli variables are used to control the expansion/shrinkage decision and the spatial- expansion/Shrinkage strength along the boundary. This model ensures the noisy label is continuous distortion of the ground truth label along the boundary, as shown in Fig. Tcl Our model also includes random flipping noise, which allows random (yet sparse) mislabels appear even regions far away from the boundary.
Based on our Markov label noise, we propose a novel algorithm to recover the true labels by removing the bias. Since correcting model bias without any reference is almost impossi- ble (Massart & Nédélec 2006), our algorithm requires a clean validation set, i.e., set of well- curated annotations, to estimate and correct the bias introduced due to label noise. We prove theoretically that only a small amount of vali- dation data needed to fully correct the bias and clean the noise. Empirically, we show that single validation image annotation is enough for the bias correction; this is quite reasonable in practice. Furthermore, we generalize our al- gorithm to an iterative method that repeatedly trains a segmentation model and corrects labels, until convergence. Since our algorithm, called Spatial Correction (SC), is separate from the DNN training process, it is agnostic to the backbone DNN architecture, and can be combined with any segmentation model. On variety of benchmarks. our method demonstrates superior performance over different state-of-the-art (SOTA) baselines. To summarize, our contribution three-folds.
(a)
(b)
(c)
Figure I: (a) Original image with true segmen- tation boundary (blue dash line). (b) Classifica- tion label noise model in segmentation context is unrealistic, where the label noise (small squares) spread allover the mask. (c) A realistic segmen- tation noise generated by our noise model. The noise is mostly about distortions of the bound- ary. A few random flippings appear at the inte- rior/exterior.
We propose Markov model for segmentation label noise. To the best of our knowledge, this is the first noise model that is tailored for segmentation task and considers spatial correlation.
We propose an algorithm to correct the Markov label noise. Although a validation set is required to combat bias, prove that the algorithm only needs small amount of validation data to fully recover the clean labels.
We extend the algorithm to an iterative approach (SC) that can handle more general label noise in various benchmarks and we show that it outperforms SOTA baselines.
2 RELATED WORK
Methods in classification label noise can be categorized into two classes, i.e., model re-calibration and data re-calibration. Model re-calibration methods focus on training robust network using given noisy labels. Some estimate noise matrix through special designs of network architec- ture (Sukhbaatar et al., 2015, Goldberger & Ren-ee 2017) or loss functions (Patrini et al.
2017 Hendrycks et al. 2018). Some design loss functions that robust label noise (Zhang &
Sabuncu| 2018 Wang et al. 2019b] Liu & Guo 2020; Lyu & Tsang 2020: Ma et al.) 2020). For example, generalized cross entropy (GCH) (Zhang & Sabuncu 2018) and symmettrc cross entropy (SCE) Wang et al., 2019b) combine both the robustness of mean absolute error and classification strength of cross entropy loss. Other methods (Xia et al. |02 Liu al. 2020 Wei et al. (022] add a regularization term to prevent the network from overfitting to noisy labels. Model re-calibration methods usually have strong assumptions and have limited performance when the noise rate is high.
Data re-calibration methods achieve SOTA performance by either selecting trustworthy data or cor- recting labels that are suspected be noise. Methods like Co-teaching (Han et al.) 2018) and (Tiang et al. 2018 Yu et al. 2019) filter out noisy labels and train the network only on clean samples. Most recently, TTanaaa et al. (2018): Zheng et al. (2020); Zhang et al. (202T) propose methods that can correct noisy labels using network predictions. Li et al. (2020) extends these methods by maintain- ing two networks and relabeling each data with a linear combination of the original label and the confidence of the peer network that takes augmented input.
Training Segmentation Models with Label Noise, Most existing methods adapt methods for clas- sification the segmentation task. Zhu et al. (2019) utilize the sample re- re-weighting technique to train a robust model by adding more weights on reliable samples. Zhang et al. (2020c) extend Co- teaching (Han al. 2018) Tri-teaching. Three networks are trained jointly, and each pair of
2

---

Published as a conference paper at ICLR 2023
networks alternatively select informative samples for the third network learning, according to the consensus and difference between their predictions. Zhang et al. (2020b) corrects pixel-wise label noise directly using network prediction, while Li et al. (2021) is based on superpixels. And Liu et al. (2022) apply early learning techniques into segmentation. All these methods fail to address the unique challenges in segmentation label noise (see Section (A.3) namely, the spatial correlation and bias. Since segmentation label noise concentrates around the boundary, methods taking every pixel independently will easily fail.
3 METHOD
We start by introducing our main intuition about each subsection before we discuss its details. In Section [.I]] aim at modelling the noise due to inexact annotations. When annotator segments an image by marking the boundary, the error annotation process resembles a random distortion around the true boundary. In this sense, the noisy segmentation boundary can be obtained by ran- domly distorting the true segmentation boundary. We model the random distortion with Markov process. Each Markov step is controlled by two parameters 01 and 02. 0₃ controls the probability of expansion/shrinkage. It has probability 81 to move towards exterior and 1 - 01 to move towards interior. 0₂ represents of the probability of marching. i.e., point on the boundary have probability Og to take step and 1 - 02 to halt. Fig. "iggBBillustten Billustrates such process. We start with the true label, go through two expansion steps and one shrinkage step. At each step, we mark the flipped boundary pixels. If 0, # 0.5, the random distortion will have a preference to expansion/shrinkaee This will result in a bias in the expected state. A DNN trained with such label noise be inevitably affected by the bias. Theoretically, this bias is challenging to be corrected by existing label noise methods, and in general by any bins-agnostic methods (Massart & Nédélec 2006). Therefore, we require a reasonably small validation set to remove the bias.
In Section 3.2 we propose a provably-correct algorithm to correct the noisy labels by removing this bias. We start by T 1. Since every pixel on the foreground/hackground boundary has the same probability to be flipped into noisy label, the expected state, i.e. the expectation of the Markov process, only has three cases, taking one step outside, taking one step inside, or staying unchanged. This indicates the relationship between the expected state and the true label can be linearized. We prove this using signed distance representation in Lemma [ If the expected state for each image is given, with only one corresponding true label, we can recover the bias in the linear equation. However, in practice, the DNN is learned to predict the expected state, and there will be an approximation error. Therefore, more validation data may be required to get a precise estimation. In Theorem [ we prove that with a fixed error and confidence level, the necessary validation set size is only O(1).
The algorithm in Scction 3.2 is designed for our Markov noise, where each point's 's moving probabil- ity only depends on its relative position on the boundary. In real-world, this probability can also be feature-dependent. To combat more general label noise, in Section 333] we extend our algorithm to correct labels iteratively based on logits. In practice, logits are the network outputs before sigmoid. Unlike distance function, logits contain feature information. A larger absolute logit value means more confident the prediction can be. Subtracting the bias from logits can move boundary points ac- cording to features, i.e., regions with large confidence move less than regions with small confidence. After correcting noisy labels, we retrain the DNN with new labels and do this iteratively until the estimated bias is small enough. We summarize our framework in Figure
Notations. We assume 2D input image X E x XC with height H. width W, and channel C, although our algorithm naturally generalizes to 3D images. Y = {0, H) x is the underlying true segmentation mask with L. classes in a one-hot manner. When training segmentation model with label noise, we are provided with a noisy training dataset D = {(X,,, Y,) Here all noisy masks are sampled from the same distribution P(Ý P(Y denotes the true label distribution. We use subscript j < W) is the index set. The scalar form X,, Yg denote the pixel value of index s and its label. For the rest of the paper, we assume binary segmentation task with Foreground (FG) and Background
to represent pixel index, where / = { [[ < < H, <
(BG) labels. Since Y is defined in one-hot manner, our formula and algorithm can be generalized to multi-class segmentation easily.
3

---

Published as a conference paper at ICLR 2023
Model Training
Label Correction
Corrected Mask
Network Prediction
Input Images
DNN
Noisy Mask
Spatial
Train
Figure 2:
Framework of our method. We train a DNN using noisy labels. The learned DNN predic- tion boundary (red dashed line) is corrected to the new boundary (black solid line). We corrected labels to re- re-train the network. The iterative algorithm can correct the noisy predictions to true labels progressively.
(a)
(b)
(c)
(d)
Figure 3: Illustration of a 3-step Markov process. (a) The true label mask, where red pixels are foreground and blue pixels are background. We mark background boundary pixels and foreground boundary pixels as 0B and OF, respectively. (b) An expansion step. Pixels marked as '1' have been flipped into foreground. The flipped pixels are randomly chosen with probability 0₂ from the ÜB pixels in (a). (c) Another expansion step by flipping pixels marked as '2' to foreground. (d) A shrinkage step. Pixels marked as '3' were foreground in (c) but are flipped into background in (d).
3.1 MODELING LABEL NOISE AS A MARKOV PROCESS
For input image X, denote the clean and noisy masks Y and Y E [0, The finite Markov process is denoted as Mc 01, 02), with T denoting the number of steps. O2 and 02 are two Bernoulli parameters denoting the annotation preference and annotation variance, respectively. To further enhance the modeling ability, we also introduce random flipping noise at regions far away from the boundary into our model. This is achieved by adding a matrix-values random noise e (Bernouli(0g)) HxW into the final step. The formal definition of (T, , 01, 02) is in Definition! 1
Denote by F, B the foreground and background masks, i.e. F and B --- the boundary operator O of F and B. Let OF F. i.e. foreground pixels adjacent to B holding value otherwise index Similarly, let 0B be the boundary mask of B. Note that sets {ŌF 1} and {aB 1} are on the opposite sides of the boundary, as shown in Fig. 3(a). For simplification, we will abuse the notation AB for both matrix and set of indices {ØB 1}. Readers will notice the difference easily. The Markov process the (-th step generates the 1-th noisy label ( based the (4- 1)-th noisy label (t-1), Denote by ((t-1) and (t- the foreground and background masks of (t-1),
Y. We define 1i.=s...ee,,.BB...... = be the boundary mask of N. the four-neighbor of
Definition (Markov Label Noise). Let (0) Bernoull and z i.i.d. (Bernoulli(02)) HxW
Y. For t 0. T let 21
(0) sid.
((+1)) (() + (t) 2 о OB(() + (z 1) ② O0(
(1)
The final output noisy label Y = (T) - Sign, where Sign (T) F(T) F. The process is denoted by (T 02)
The random variable ( determines whether the noise is obtained by expanding or shrinking the boundary in each step 1. If expansion, 2(0) 1, the is taken by flipping the background
4

---

Published as a conference paper at ICLR 2023
boundary pixels (B(t-)) with probability 02. This is encoded in the second Bernoulli Z OB(), where ( an indication matrix restrict the flipping only happens among background boundary pixels. Similarly, if shrinkage, 2 = 0, a pixel in ( has probability 02 to be flipped into background. Fig. 3 shows example with 3 steps, corresponding expansion, expansion and shrinkage.
3.2 LABEL CORRECTION BY REMOVING BIAS
Suppose the Markov noise has underlying posterior XXY which impossible get general explicit form due to the progressive spatial dependency. However, we can study the Bayes classifier e(X) = arg maxy P.Y.X). For fixed image X, we take Y ~ X) as random variable. Then Y| and e(X) have,
x(x) L3 ||
where [|[[]] means for each index s, E [20.((s) 1, if [E]. 2 0.5, otherwise, Note that E B is probability map while [| is its (Bayes) prediction. Therefore, to recover the true Bayes classifier c(X) is to build an equation between E and E|y]. we iterate our Markov model Equation over t O.... T plus the random flipping term and take expectation on both sides,
T-1
T-1
= E Y + 0102
E 11 + (01 1)02
Σ
E[OF + +gE.[iignl
(3)
Equation 3 defines the bias between E Y| and [ But the ambiguous representation of ((aB()) makes it difficult to simplify the bias. In order to analyze it, we need to transform the explicit bound- ary representation 0B into implicit function. For domain Ω, define its interface (boundary) as SS, its interior and exterior as Q and Ωt,, respectively. An implicit interface representation defines the interface as the isocontour of some function 6(.). Typically, the interface is defined as an = {s (s)) = 0}.
A straightforward implicit function is the signed distance function. Following Osher & Fedkiw (2003). a distance function d(s) is defined as d(s) = minteaal - 4)), implying that d(s) = On. A signed distance function an implicit function defined by [p(s) d(s) for all E Ω2 s.t. ((s) d(s) = 0 for a2, o(s) d(s) for s E Q+ and ((s) -d(s) for E Ω For domain of grid points, we define the distance function d(s) by the shortest path length from to NR. But note that On does not exist among image indices. It lies between ÖB and OF. The corresponding definition for image indices is as follows.
Definition 2 (Signed Distance Function). For an Image X with index set 1, a graph G(S, E) is constructed by S = I and m = {s + t E I,t E N.). where N, is the four-neighhor of index Then the distance d(ss, t) is defined by the length of the shortest path between and t in graph G. The distance function is defined by
d(s) = mmincoB d(s, + 11, E
(4)
min d(s, 1) + 1, otherwise.
The signed distance function (s)) is then defined by Φ(s) = d(s), if E B. Otherwise, d(s) = -(ss).
We start from the case T = 1. With the signed distance representation, we have the following lemma.
Lemma 1. o is the signed distance function defined for ef c(X). 1f0g < 0.5, then
= [ [0,02]20.5 - 1,
if 2 0.5, otherwise.
(5)
Φ [1+ 0,02 02) <0.5 +1
Here is the signed distance function of e(X) when 1.
With the signed distance function, we can explicitly define the bias by A = Evero - ds), where [7] is the cardinality of the index set, also the image size. If ē(X) is perfectly learned, the
5

---

Published as a conference paper at ICLR 2023
difference between @ and i.e. A can be estimated with small clean validation set, even if there is only one image. Then the original function & can be recovered among training set by & ~ + A. Then c(X) can be simply obtained by [ő] co- However, in practice, the classifier learned, denoted by )(X), is different from the noisy Bayes optimal x(X). This will lead an error to Ø, signed distance function of ě(X). for which more validation data may be required. The naive algorithm is,
1. Bias estimation. Train DNN with noisy labels. Given clean validation set bias estimated by
the
0% - --"). SET
(6)
2. Label correction. Correct training labels by (X) = [0' where ø' - Å. Then retrain the network using corrected labels.
In Theorem we prove that with small validation size, the above algorithm can recover the true label with bounded error.
Theorem 1. If 3 Eo
0, E1
0. s.t. Ex sup, (²(X.) -(xx)1 S Eo and
supx sup, ((xx) - o(x,) s hold for the learned classifier é(X), then Ve > Eo and for
a fixed confidence level 0 < a S 1, with
E log
2/11
V Z 2( - 50)2
a
number of clean samples (Xo, vol 1= o' can be recovered within E + error with probability at least 1 - a, i.e. P(Ex [sup, (6'(X,) - <(x.]l S & + Eo) 2 a.
Proofs of Lemma and Theorem are provided in Appendix. The theorem shows that for fixed error level E and a lixed confidence level a, the validation size V required is a constant, logarithm of the image size. Note that Eg the mean model error among images and E1 is the supremum model eтror among images. It holds naturally that Eo I E1. E1 is to constrain the model prediction variance on different images. Eo = E1, the model prediction quality is similar over images. Even in the worst case, where model cannot predict any right label, is still bounded by the image size [T)
3.3 ITERATIVE LABEL CORRECTION
In this section, we extend our algorithm to more general label noise. In our Markov model, all points the boundary are moving with the probability along the normal direction of signed distance function. The boundary in real-world noise, however, could have feature moving probability. We achieve this by using logit function representation. For an image X, its logit function f(X) is defined by EY segmentation label. Note that fXX is positive interior 5 and negative in Ω+, so the implicir function is defined by (XX). The bias estimation step remains the same. But at the label correction step, apply the bias f(X) instead of J. In practice, f(X) is the model output before the sigmoid function. The label correction step by logit function is
f(X). where a(.) is a sigmoid function and Y is the
- = + Acxp
(8)
2(7
is the bias adapted to logit function and the exponential term is a decay function to constrain the bias around & = O. And the decay factor is 7', where 0 < I 1 is a hyper-parameter, usually set to be 1. The bias is defined by X = inf flos. if Å > 0, and > = sup f /'ás if Å < 0. fla is function f restricted on domain f2. Although the bias is decayed with the same scale over the distance transform, the gradient logit function, however, varies along the normal direction. Therefore, Equation 8 actually moves points on the interface with different degrees. Points with smaller absolute gradient moves more. Since the algorithm corrects noisy labels spatially correlated to the boundary, refer to it as Spatial Correction (SC). We present our SC in Algorithm 1. In practice, the terminate after only iteration.
iterative approach y is usually set to be 1, and the algorithm can

---

Published as a conference paper at ICLR 2023
Algorithm Spatial Correction
Input: Noisy training dataset D, small clean validation dataset V. and hyper-parameter 7.
Output: A robust DNN f trained with denoised dataset.
Train a DNN with D.
Predict labels on V uuinn f, and estimate Å by Equation 6
while / 2 1 do
Replace training labels with [["()) where ('(X) calculated by Equation
Re-train f with corrected labels.
Predict labels on V using new f. and estimate Á by Equation 6 end while return f.
(a) JSRT SE.
(b) ISIC 2017 Ss.
(c) Cityscapes.
Figure 4: Examples of synthetic and real-world noise. In each image, blue line is the true segmen- tation boundary, and all other colors are corresponding noisy boundaries. We removed the random flipping noise in visualization focus the boundary.
4 EXPERIMENTS
In this section, we provide an extensive evaluation of our approach with multiple datasets and noise settings. We also show in ablation studies, that our method is robust high noise level and the required clean validation set size can be extremely small.
4.1 DATASETS AND IMPLEMENTATION DETAILS
Synthetic noise settings. We use three public medical image datasets, JSRT dataset (Shiraishi
et al. 2000), ISIC 2017 dataset (Codella et al. 2017), and Brats 2020 dataset (Menze et al.) 2015
Bakas al. 2017b 2018 2017a). JSRT contains 247 images for chest CT and three types of organ structures: lung, heart and clavicle, all with clean ground-truth labels (Juhász et al. 2010).
We randomly split the data into training (148 images), validation (24 images), and test (75 images) subsets. ISIC 2017 is a skin lesion segmentation dataset with 2000 training, 150 validation, 600
test images. Following standard practice Zhu et al. (2019): Zhang et al. (2020b); Li al. (202I).
we resize all images in both datasets to 256 x 256 in resolution. Brats 2020 is a brain tumor 3D segmentation dataset with 369 training volumes. Since only training labels are accessible, we randomly split these 369 volumes into training (200 volumes), validation (19 volumes) and test (150 volumes) subsets. resize all volumes 64 x 128 128 resolution.
For each of these three datasets, we use three noise settings, denoted by S5. Ss and S SØ and Ss are two settings synthesized by our Markov process with 01 0.5 (expansion) and 0, < 0.5 (shrinkage), respectively. Figure 4 shows examples of our synthesized label noise. We also include the mix of random dilation and erosion noise SN used by previous work (Zhu et al., 2019 Zhang al. 2020b a). This is achieved by randomly dilate erode mask with number of pixels. Note that our Markov label noise can theoretically include this type of noise by setting 0₂ = 0.5. Detailed parameters for these settings are provided in the Appendix.
We use simple U-Net as our backbone network for JSRT and ISIC 2017 datasets and a 3D-UNet for Brats 2020 dataset. The hyper-parameter is set to be and total iteration is 1.

---

Published as a conference paper at ICLR 2023
Label correction results for Brats 2020 datasets with Sg (Left) and Ss (Right) settings. From top to bottom, the rows are sample slices with noisy masks, with true masks, and with corrected masks, respectively.
Figure 5:
Real-world label noise. To evaluate with real-world label noise is challenging. We are not aware of any public medical image segmentation dataset that has both true labels and noisy labels from human annotators. Therefore,
multi-annotator dataset, LIDC-IDRI dataset (Armato III et al. 2015| Armato et al., 2011 Clark et al. (013)) and the coarse segmentation in a vision dataset, Cityscapes (Cordts al. 2016). The LIDC-IDRI dataset consists of 1018 3D thorax CT scans where four radiologists have annotated multiple lung nodules in each scan. The dataset was annotated by 12 radiologists, and it is not possible voting as the true labels and the union of four annotations as noisy labels. We process and split the data exactly the same way as Kohl al. (2018). Cityscapes dataset contains 5000 finely annotated images along with coarse segmentation by human annotators that we use the "noisy label".. We only focus on the 'car' class because (1) cars are popular objects and are frequently included in images; (2) the coarse annotation of cars is very similar to noisy annotation in medical imaging they are reasonable distortions of the clean label without changing the topology. See Figure 4c for an example. The detailed settings of LIDC-IDRI and Cityscapes can be found in Appendix A.2.1
match an annotation to an expert. We use the majority
Baselines. We compare the proposed SC with SOTA learning-winn label-noise methods from both classification (GCE (Zhang & Sabuneu 2018). SCE (Wang et al.] 2019b), CT+ (Yu et al. 2019)
ELR (Liu et al. 2.200. CDR (Xia et al. 2021) and segmentation contexts (QAM (Zhu et al. 2019
CLE Zhang et al. (020bb). Technical details of these baseline methods are provided in Appendix A.2.2 Our method requires small clean validation set whereas most baselines do not. For a fair comparison, we also use the clean validation set to strengthen the baselines. In particular, we pretrain the baselines using the clean validation dataset, and then add the validation images and their clean labels into the training set. Note that our method (SC) only trained on the original noisy trianing set; it only uses the validation set to estimate bias.
4.2 RESULTS
shows the segmentation results of different methods with synthetic noisy label settings on JSRT ISIC 2017 and Brats 2020 dataset. Note that QAM cannot be applied to Brats 2020 dataset because their network is designed for 2D only. We compare DICE score (DSC) on testing sets (against the clean labels). For each setting, we train 5 different models, and report the mean DSC and standard deviation. In Sg and Ss. where biases show up in noisy labels, the proposed method outperforms the baselines by big leap in total case. The compared methods, however, only work when little bias is included, like SM- SM is equivalent to setting 0 = 0.5 in Markov model, resulting in
Table
0. We also test the proposed method on real-world label noise, results shows in Table 2 Figure shows examples of label correction results. We provide more qualitative results in the Appendix A.4
4.3 ABLATION STUDY
Increasing Noise Level. Our proposed method is robust even when the noise level is high. In Figure 6aa we compare the prposed SC with SCE and ELR. We increase the synthetic noise level on JSR7 dataset by increasing the step T in our Markov model, while keep the same 81 and 0₂ (details and illustrations in the supplementary material). Results show our method is still robust even under extreme noise level, while the performance of GCE and ELR drops rapidly as the noise level increases.
8

---

Published as a conference paper at ICLR 2023
Table 1: Mean DSC (in percent) and standard deviation for five models trained on three noisy settings. Method with best mean DSC is highlighted for each noise setting.
JSRT
Method
ISIC 2017
Brats 2020
Lung
Teart
Clavicle
Total
GCE
79.43
71.70.11000 0.36
58.36
0.14
69.90
0.12
73.12 1 0.27
68.07 1 0.78
SCE
81.03
1.98
79.24 2.90
73.38 + 1.98
77.88 + 1.45
74.06
+ 0.92
68.04 + 1.71
CT+
82.08
0.06
66.37 0.08
44.63 1 0.10
64.43
1 0.07
68.65
1 0.20
70.69 1 0.72
ELR
79.45
79.69 0.56
79.37 + 0.18
79.51
0.16
72.64 + 0.99
67.56 + 0.82
Se
CDR
78.77
73.64 1.03
63.77
+ 0.93
72.06
1: 0.30
T2.76
0.95
66,95 +: 2.23
QAM
0.445
78.63 78.63 0.20
78.71
0.14
78.63
1: 0.14
71.32
+: 0.45
CLE
0.88
78.19.10.66
45.75
2.48
66.14
1.21 72.95
65.93
SC(Our))
2.86
0.96
78.11 1.35
86.04
1.76
0.63
77.78 0.85
GCE
0.40
90.84 0.17
71.33 0.36
83.56 = 0.19
52.44 3.47
47.65 = 1.42
SCE
6..04+1.771
75.83 + 1.59
82.49 + 0.30
78.67 + 1.38
53.03 + 3.34
47.01 + 2.11
CT+
93.55
84.11 0.14
53.86 0.17
77.18
67.55 + 0.53
71.69 - 0.01
Ss
ELR
0...5
11.15.1.1
71.76 1: 0.25
72.35 1: 0.37
50.21 1 3.41
52.36 1 5.48
CDR
23.00
2.13
75.53
80.74
50.87 1 1.56
51.51 7.98
QAM
72.76
68.71
1.51
70.68 0.80
70.72 0.91
52.95 + 3.10
CLE
83.86 + 1.30
50.56 2.83
72.13 1.30
54.76
1.15
49.81 9.63
SC(Ours)
0.100
92.05
0.48
2.02 75.78
87.41 0.81
75.97
1.73
72.71
1.20
GCE
86.51 0.39
79.74 0.93
55.83
= 0.74
74.03 = 0.41
77.98 + 0.33
73.10 = 0.38
SCE
86.73
86.18 1.21
70.35
4.03
81.09 1 1.20
79.00
1 0.72
70.52
2.67
CT+
0.22
73.35 0.08
37.50 0.41
0.211
74.97
0.39
71.34
0.18
ELR
87.90
89.09
1.93 71.28
82.76
2.91 77.39
4.71
SM
CDR
85.87 1.39
82.84 1.81
63.75
4.46
77.48 1.55
79.16 1.07
74.21
1.54
QAM
85,64 1.26
87.79 1.15
67.43
5.73
80.29 1 2.49
76.26 1 0.72
CLE
86.85 + 0.84
83.90 + 0.93
57.40
2.52
76.05
1.04
77.1 + 1.55
75.03
0.77
SC(Ours)
91.31
0.62
87.14 2.15
72.01
1.19
83.55
0.93
79.44 1 0.65
72.89 1 0.98
Table 2: Mean DSC (in percent) and standard deviation for five models trained on Cityscapes datasets and LIDC-IDRI dataset. Method with best mean DSC is highlighted.
Dataset
GCE
SCE
CT+
ELR
CDR
QAM
CLE
SC(Ours)
Gitorapey
72.02 = 0.14
76.35 0.13
72.79 + 0.14
73 0.17
71:26
0.16
71.33 0.14
82.30 + 0.10 49.79 0.39
74:41 00.12
LJOC-IDRI
42.56 + 0.58
50.40 + 0.15
40.25 + 0.11
49.76 0.40
49.41
0.79
43.56 + 0.29
53.72 + 1.70
01-
82-
Dico/%
Dice(
00-
Ours
76-
ELR
ELR
55
350
Markov Step
Validation Size
(a) Increasing noise level.
(b) Decreasing validation size.
Figure 6: Ablation study.
Decreasing Validation Size. this experiment, we show that SC works well even with an extremely small clean validation set. Following setting SM of JSKT dataset, we shrink the validation set from 24 18. 12, 6, 1, respectively. We compare the performance with SCE and ELR. Results in Figure 6bb show that our proposed method still works well when the validation size is extremely small, even if only one clean sample is provided.
CONCLUSION
In this paper, proposed Markov process to model segmentation label noise. Targeting such label noise model, we proposed a label correction method to recover true labels progressively. We provide theoretical guarantees of the correctness of a conceptual algorithm and relax it into more practical algorithm, called SC. Our experiments show significant improvements over existing approaches on both synthetic and real-world label noise.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors acknowledge the National Cancer Institute and the Foundation for the National Institutes of Health, and their critical role in the creation of the free publicly available LIDC/IDRI Database used in this study.
This research of Jiachen Yao and Chao Chen was partly supported by NSF CCF-2144901. The reported research of Prateek Prasanna was partly supported by NIH 1R2ICA258493-01AI. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Mayank Goswami would like acknowledge support from US National Science Foundation (NSF) grant CCF-1910873.
REFERENCES
SG 3rd Armato, G McLennan, and al. The lung image database consortium (lide) and image database resource initiative (idri): A completed reference database of lung nodules on et scans. Medical Physies, 2011.
S. G. Armato III, G. McLennan, and et al. Data from lidc-idri [data set]. The Cancer Imaging Archive, 2015.
Devansh Arpit, Stanistaw Jastrzundefinedbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julics closer look at memorization deep networks. In ICML, 2017.
S Bakas, H Akbari, A Sotiras, M Bilello, M Rozycki, J Kirby, J nad Freymann, K Farahani, and C Davatzikos. Segmentation labels and radiomic features for the pre-operative scans of the tega- gbm collection. The Cancer Imaging Archive, 2017a.
Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Sci Data, 2017b.
Spyridon Bakas, Mauricio Reyes, and al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge, 2018.
K. Clark, B. Vendt, and et al. The cancer imaging archive (tcia): Maintaining and operating a public information repository. Journal of Digital Imaging, 2013.
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern Skin lesion analysis toward melanoma detection: A challenge at the 2017 international sympo- sium biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). ArXiv, 1rXiv:1710.005:: 10.05006 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Danna Gurari, Diane Theriault, Mehrnoosh Sameki, Brett Isenberg, Tuan A. Pham, Alberto Pur- wada, Patricia Solski, Matthew Walker, Chentian Zhang, Joyce Y. Wong, and Margrit Betke. How to collect segmentations for biomedical images? a benchmark evaluating the performance of experts, crowdsourced non-experts, and algorithms. In WACV, 2015.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, 2018.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett "ods.,,Advannee in Neural Information Processing Systems. Curran Associates, Inc., 2018
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICMI,, 2018.
Leo Joskowicz, D. Cohen, N. Caplan, and Jacob Sosna. Inter-observer variability of manual contour delineation of structures in ct. European Radiology, 2018.
Juhász, Á Horváth, L. Nikházy, and Horváth. Segmentation of anatomical structures on chest radiographs. In XII Mediterranean Conference on Medical and Biological Engineering and Com- puting 2010. Springer, 2010.
Simon Kohl, Bernardino Komera-Paredess, Clemens Meyer, Jeffrey De Fauw, Joseph R. Ledsam, Klaus Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A prob- abilistic u-net for segmentation of ambiguous images. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bisa-BBancch, and R. Garnett (eds.), Advances Neural Information Processing Systems. Curran Associates, Inc., 2018
Mare Kohli, Ronald Summers, and Jr Geis. Medical image data and datasets in the era of machine Bearnin--whitppppe from the 2016 c-mimi meeting dataset session. Journal of digital imaging, 2017.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In /CLR, 2020.
Shuailin Li, Zhitong Gao, and Xuming He. Superpixel-guided iterative learning from noisy labels for medical image segmentation. In MICCAI, 2021.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernande Early-learning regularization prevents memorization of noisy labels. NeurlPS, 2020.
Sheng Liu, Kangning Liu, Weicheng Zhu, Yiqiu Shen, and Carlos Graande Adaptive early-learning correction for segmentation from noisy annotations. CVPR 2022, 2022.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. ArXiv, abs/1910 fts/1910.0321,, 2020.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICML, 2020.
Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. The Annals of Statistics, 2006.
Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Sellpathy-Cramer Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). TMI, 2014.
Bjoemn H Menze, Andras Jakab, and et al. The multimodal brain tumor image segmentation bench- mark (brats). TMI. 2015.
Reinhold Nafe, Kea Franz, Wolfgang Schlote, and Berthold Schneider. Morphology of tumor cell nuclei significantly related with survival time of patients with glioblastomas. Clinical cancer research, 2005.
Stanley J. Osher and Ronald Fedkiw. Level set methods and dynamic implicit surfaces. Applied mathematical sciences. Springer, 2003.
11

---

Published as a conference paper at ICLR 2023
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: loss correction approach. In CVPR, 2017.
Junji Shiraishi, Shigehiko Katsuragawa, Junpei Ikezoe, Tsuneo Matsumoto, Takeshi Kobayashi, Ken-ichi Komatsu, Mitate Matsui, Hiroshi Fujita, Yoshie Kodera, and Kunio Doi. Development of a digital image database for chest radiographs with and without a lung nodule. American Journal of Roentgenology. 2000.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. In ICLR, 2015.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, 2018.
Feifan Wang, Runzhou Jiang, Liqin Zheng, Chun Meng, and Bharat Biswal. 3d u-net based brain tumor segmentation and survival days prediction. MICCAI Brainlesion Workshop. Springer, 2019a.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019b.
Hongxin Wei, Lue Tao, RENCHUNZI XIE, and Bo An. Open-set label noise can improve robustness against inherent label noise. In NeurlPS, 2021.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2021.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? ICML, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR,
2016.
Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Joseph Jacob, Olga Ciccarelli, Frederik Barkhof, and Daniel C. Alexander. Disentangling human eπror from the ground truth in segmentation of medical images. NeurIPS, 2020a.
Minqing Zhang, Jiantao Gao, Zhen Lyu, Weibing Zhao, Qin Wang, Weizhen Ding, Shuidi Wang, Zhuguo Li, and Shuguang Cui. Characterizing label emror: Confident learning for noisy-labeled image segmentation. In MICCAI, 2020b.
Tianwei Zhang, Lequan Yu, Na Hu, Su Lv, and Shi Gu. Robust medical image segmentation from non-expert annotations with tri-network. In MICCAI, 2020c.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami. and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In ICLR, 2021.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS. Curran Associates Inc., 2018
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-hounded correction of noisy labels. In ICML., 2020.
Haidong Zhu, Jialin Shi, and Ji Wu. Pick-and-learm Automatic quality evaluation for noisy-labeled image segmentation, 2019.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOFS
Lemma 1. Φ the signed distance function defined for domain [ΕY] 20.5. If03 < 0.5, then
Φ 00.02 1,
if0 2 0.5,
(A.1)
④ + [ + 0,02 k0.5 +1
otherwise.
Here Φ is the signed distance function of [EY]20, when =
Proof. When = 1, the given is deterministic, we have
E Y + +10gE[BB]] + (0: 1)0,E(OF] + [Sig(
(A.2)
Recall that Sign
B
B F(T) o F, indicating Sign, 0 E OF U UBB Therefore,
0102,
E 0B
E[Y,]
1 0,02 O2, E OF
(A.3)
Yg + 03, otherwise
Since Y, binary, and 0.5, [Y. + 02/20.5 Ys. Therefore,
10102|20.51
s ÒB [1 + 0,000 02/20.5 8 OF otherwise.
((X.)
(A.4)
Y.:
We claim the following three facts.
(1) 0 02 2 0.5 only if 0 2 0.5.
This is true because if 0 0.5, 0,0g 0.5 since 0g S 1.
(2) 1 + 0,0 0g < 0.5 only 0 < 0.5
0.502 2 0.5. I 2 0.5, then + (0) 1)0- 2
(3) 0,02 2 0.5 and +0,02 02 - 02 < 0.5 are mutual exclusive.
First we notice when 0,0, 2 0.5, 1 0,02- 0g 1.5 02 0.5. Then when + 0,02 02 0.5, 0,02- < 02 0.5 < 0.5.
The first facts separate the cases in Equation A.1 And according to fact (3), either ÖB is flipped into foreground or OF is flipped into background. In the former case, expansion happens because 0. > 0.5, and OB becomes OF, leading to I = o [0,02] 1. Opposite happens in the latter case. Therefore,
Φ 10,02]205 - 1,
if 2 0.5. otherwise.
(A.5)
p + 11+000 021
Proof done.
Theorem 1. 15, > 0, s.f.
Ex sup (x(x.) - x(x,)1 $ SED,
(A.6)
and
sup x
sup (²(X.) - -x(x)) S1,
(A.7)
hold for the learned classifier é( X), then VE Eu and for fixed confidence level a 1, with
2
V >
2( 50) log
(A.8)
number of clean samples (x,,y, y. V. o' can be recovered within E + Eo error with probability at least 1 a. i.e. P(Ex (sup, 10 (6'(x)) o(X.)l E0)
13

---

Published as a conference paper at ICLR 2023
Proof. To bound the label correction error, we first aim to bound the bias estimation error. We try to prove that
P(IÁ
E) a
(A.9)
Recall that the definition of Á is
(A.10)
and A is the true bias that is the same for every image X. Hence,
-7)
(A.11)
D=l
SEI
Take Equation 1.10amm and Equation A.I1
into the LHS of Inequality A.9
1
P(IÁ - 2
)%)
28
(A.12)
1
P
30
21
7
Therefore, to prove A.9 is equivalent to proving
18
3
2
Sa.
(A.13)
Given index the absolute error bounded by the image size, i.e. bound is provided for the following probability.
10y
is i...d over different images X And the error is E1 According to Hoeffding's inequality, lower
10:
S
2V E
P
3
Ex
|0% 6:1|
2
s
exp
(A.14)
By the given model enror A.6
Vs E I,
Ex
[5
$ Ex
sup
10= 0z
$ E0.
(A.15)
Combine Inequality A.14
and A.15
2V 60)
P
0 351 2
< 2 exp
(A.16)
oî
Observing the LHS of A3 and A.16 the difference that random variable A.13 the average error among specific group of indices, while 16 holds for arbitrary index. If we iterate A. A. oveer the index set, then for a group of indices, A3 13.13 naturally holds. other words,
1 Σ 1
5: -
-
10
S SET
Σ
10
3.
26
(A.17)
V
Hence,
Σ
Σ
18 कभ 23
2E
(A.18)
2V 80)² 2
$ 2 exp
To obtain A.13 let
2V(E - 20
2 exp
Sa,
(A.19)
14

---

Published as a conference paper at ICLR 2023
and if
V oi (e ED)
2|1|
log
(A.20)
then
P(IÁ A 2
(A.21)
Next we are about bounding the label correction error. Note that @ = Φ A, and our label correction step indicates that ø' Å. Then we have,
sup ((((x.) o(x.)) + (A إاه
Ex
sup jØ' (x,))) oxx)) = Ex
Ex sup ((((.)) 3(X.))| +1A Á
(A.22)
Eo A ÅV.
Therefore,
P( A
22 P(Iá - A 3O 60)
(A.23)
P Ex sup | (X,) e(Xx))l 2 2 -
According to A.2I
we proved
sup (x'().) ) - x(x.)| 2 2 sa.
P
Ex
(A.24)
A.2 IMPLEMENTATION DETAILS
All the experiments are done on one NVIDIA GTX 1080 GPU (12G Memory) with batch size of 2. For JSRT, we use a learning rate of 0.05. At 750 iter, we decrease the learning rate by a factor of
10, with total 1.5K iterations. For ISIC 2017 and Cityscapes, the learning rate also set to be 0.05, and decay is done every 4K iterations for ISIC 2017 and 3. 3.1K iterations for Cityscapes, by factor 2f The model converges at around 10K iterations.
A.2.1 NOISE AND DATASET SETTINGS.
Synthetic Noise Setting. The synthetic noise follows parameters in Table A. Se stands for expansion setting and Sş standing for shrinkage setting. And for the ablation study of noise level, keep the same 0, 02 in Sg of JSRT dataset, and increase T from (180, 180, 100) (for lung, heart, clavicle) to (250, 250, 170), (300, 300, 220). (350, 350, 270), (400, 400, 320), respectively. 0g is set 0.1 for all settings. To improve the smoothness, we also use Gaussian filter before the random flipping. An example for increasing noise level of heart class is shown in Fig, A.
Table A.1: Label noise settings on two synthetic datasets. M(T, 0,, 02) stands for the proposed multi-step Markov process.
Noise
JSRT Heart
ISIC 2017
Brats 2020
Setting
Lung
Clavicle
M(180, 0.7, 0.03)
1180 0.03)
M(100, 0.7, 0.03)
(200, 0.8, 0.05)
M M 0.7, 0.05) (S0, 0.3, 0.05)
Sy
M2200, 0.3, 0.05)
[200, 0.3, 0.05)
M 0.3, 0.05)
(200, 0.2, 0.05)
Dataset Settings. For Brats 2020 dataset, we merge all three classes into a single class. The reason that some classes are so few in volumes that most of them could vanish when we create the synthetic noisy labels. For LIDC-IDRI dataset, we follow the pre-processing of Kohl et al. (2018) by extracting 2D 128 x 128 slices centred around the annotated nodules, resulting in 8843 images in the training set, 1993 images in the validation set and 1980 images in the test set. We set y 1 in our algorithm and it terminates after iteration. For Citysacpes dataset, since the test labels are not publicly available, use the validation set test set. We resize all images to 256 512 and remove
15

---

Published as a conference paper at ICLR 2023
T=180
T=250
T=300
T=350
T=400
Figure A.I: The increasing noise level in 'heart' class. Red line is noisy boundary and blue line is true boundary.
(a) Equal dilation.
(b) Our Sæ noise.
(c) Equal erosion.
(d) Our Ss noise.
Figure A.2: Compare equal dilation (a) and equal erosion (c) with Markov noise (b) and (d) gener-
ated by our Markov model.
images whose coarse masks have few no car labels. This gives 2086 training images and 350 test images. We further randomly split the training images into 1986 training images (with noisy label) and 100 validation images (with clean label). We also use simple U-Net our backbone network. We conservatively choose a small y = 0.4, and correct label noise progressively. The algorithm terminates after 2 iterations.
A.2.2 BASELINES.
We compare the proposed SC with current SOTA methods from both classification context and segmentation context: (I)GCE Zhang & Sabuncu 2018) trains the deep neural networks with generalized cross entropy loss handle noisy labels. The hyperparameters k, this work are set to be 0.5 and 0.8, respectively. (2)SCE Wang et al. (2019b) combines the cross entropy and reverse cross entropy (RCE) into a single noise robust loss. The hyperparameters a and 3 are set to be 1.0 and 0.5 for JSRT and ISIC 2017 dataset, and = 1.0, 3 0.5 for Cityscapes dataset. (3)CT+ Yu et al. (2019) utilizes two networks and selects small-loss instances for cross training. treat each pixel noise rate т is estimated with the clean validation set. (4)ELR Liu et al. (2020) utilizes the early stopping technique into regularization term prevent the network from memorizing noisy labels. The hyperparameters X and 3 are set to be and 0.8, respectively. (5)QAM Zhu et al. (2019) re- weights samples with quality awareness module (QAM), trained together with the segmentation model. The outputs of QAM are the weights for each image in the loss function. (6)CLE Zhang al.) (2020b) leverages confident learning correct noisy labels based on the network prediction. Then train robust network with corrected labels.
To employ this method into segmentation,
instance. The prior estimated
A.3 ILLUSTRATIVE EXPERIMENTS
Network predictions can be over confident if bias shows up in training labels. Some works (Zhang et al.) 2020b Li et al., 202T choose to trust the network prediction probability to correct label noises, i.c. they believe predictions with small confidence is likely to be wrong while pixels with large confidence tend to be correctly predicted. However, the network can fit to noisy labels quickly and be overconfident when trained with biased noisy labels. This phenomenon is also observed by ZZhang et al. (2016). In A.3]) we train a network with dilated noises and show its prediction probability map, i.e. the output after sigmoid. The red pixel is predicted as foreground
16

---

Published as a conference paper at ICLR 2023
with high probability 0.9994874, whereas it is actually in background. Therefore, methods based on trusting the network predictions cannot correct this label because the network is over confident. And since the network can fit to noise rapidly, early learning techniques also cannot eliminate this bias. Our method works because we do not trust the network prediction. Instead, we compare the clean label in validation prediction. We also show how the prediction probability changes while training the model in Fig- ure Ab. It shows that the model can over confident quickly while training. So methods that
and estimate the bias. We then eliminate this bias in the training
employ early learning techniques (Liu et al. 2020 work under biased noisy labels.
Arpit et al. 2017
Liu et al. 2022) are hard to
03
0.
67
08
(b) The probability of the red point while training.
(a) Prediction probability.
Figure A.3: A model trained with biased labels can be over confident. The model is trained with dilated noises. In the left figure, the blue line the true segmentation boundary, and the red point is the selected pixel where it is supposed to be background (in the true label) but is predicted as foreground. The right figure shows how the network prediction on the red point fits to noise along training.
Noisy annotations with random inner holes. Our Markov noise model assumes the spatial bias occurs around the true boundary, but it does not reject other random noises like inner holes. Fig- ure A.4 shows an expanded noise with low-frequency inner holes (Ist row), and our method can correct this hole after iterations.
Figure A.4: A model trained with noisy labels with random inner holes (first row). The corrected
label by the proposed method after iteration (second row) and 2 iterations (third row).
17

---

Published as a conference paper at ICLR 2023
Noisy
CLE
Figure A.5: Label correction results from the baseline CLE and our method SC. Red lines are corrected boundaries, and blue lines are true boundaries.
A.4 QUALITATIVE RESULTS
We provide qualitative results for prediction on test images in Fig. A.6
An6 and Fig. A.7 as two sub parts.
We also show the label correction training images in Fig, A.5
A.S EXTENTED NOTATION
Although our notation is for 2D images, our method naturally generalizes to 3D. For 3D input X xxxC define the spatial correlation 3D input, we just need redefine the neighbor N, as the six-neighbor, i.e. the neur-neigh in the same slice and up and down pixel in adjacent slices. general, our spatial correlation can be defined for images of arbitrary dimension.
18

---

Published as a conference paper at ICLR 2023
(a) U-Net trained on clean labels.
(b) GCE
(c) SCE
(d) CT+
Figure A.6: Qualitative results of different model predictions part one. Blue lines are true bound- aries. All other colors are corresponding prediction boundaries.
19

---

Published as a conference paper at ICLR 2023
(a) ELR
(b) QAM
(c) CLE
(d) Ours
Figure A.7: Qualitative results of different model predictions part two. Blue lines are true bound- aries. All other colors are corresponding prediction boundaries.
20


Reviews:
[{"review_id": "hDgVBWfV4zd", "reviewer": "Reviewer_VNP8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a label correction method to tackle the label noise in the segmentation task.  They propose an algorithm to correct the Markov label noise.\n", "summary_of_the_review": "This paper proposed a Markov process to model segmentation label noise. The issue that this paper focuses on are important but not very attractive. The method needs to be extended for better tackle the problem of the 2D/3D segmentation label noise. Whether it is more convincing to experiment with real data labeled by different intern doctors. More specifically,\n1. It is strongly recommended that the input be changed to 3D voxel segmentation to match SC.\n2. The results of the Brats 2020 dataset should show multi-class results.  Is this method capable of handling the multi-class segmentation label noise problem?\n3. Fig.3 is hard to follow with the 3-step Markov process.", "strength_and_weaknesses": "Strength: \nThe Markov model for segmentation label noise considers spatial correlation.\n\nWeakness:\nThe spatial correlation proposed by the authors contradicts the input 2D image. Whether the authors consider three-dimensional spatial correlations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Oas-eVhtDs", "reviewer": "Reviewer_VpVv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method for learning image segmentation from noisy annotation. The noise model proposed models human annotation errors that are spatially correlated and dependent on local image features. Results of the approach is compared to a wide range of prior work using artificially added noise in three different datasets and on a single dataset with real noise.", "summary_of_the_review": "I have put the paper marginally below the acceptance threshold. I do like the paper and the approach, but I would have expected a little bit more in terms of showing that it works with real data.\n", "strength_and_weaknesses": "Strengths\n------------\n- The approach addresses the important and relevant problem of learning to segment from noisy labels.\n\n- The proposed method seems relatively simple and the ideas underlying it makes sense.\n\n- The approach is compared to a wide range of relatively recent prior work and shows superior results.\n\nWeaknesses\n----------\n- Sentences in abstract seem condenced to the point where some meaning is lost.\n\n- Table 1 list Dice scores of the model along with other methods for learning with noisy labels in datasets with artificial noise added. Noise is added according to the noise models that is also used by the proposed approach to remove the noisy labels. While this is of course a nice sanity check of the proposed approach, it is perhaps not surprising that the proposed approach would excel at this. I would suggest the authors discuss the limitations of this experiment more clearly.\n\n- Only a single experiment is done with real noisy labels. I would have appreciated more experimentation on real data. Noisy labels in segmentation is generally the rule rather than the exception, particularly with medical images. I think i\nt would have greatly strengthened the manuscript if the authors had made a greater effort to show their method can help to solve this problem.\n\n- Code is not provided, which could hurt reproducibility.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JbF6Jh0NnP6", "reviewer": "Reviewer_LKDX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach to correct noise annotation for segmentations by making use of spatially-aware operations about flipping labels on the boundary of the segmentation. The proposed method uses clean validation data to calculate the bias of the predicted segmentations and uses the specific noise model together with the inferred noise parameters to correct the predictions of the trained segmentation model. This process can be applied iteratively until the predictions are unbiased. The paper compares the proposed method on three medical imaging datasets (ISIC, JSRT and BRATS) as well as the cityscapes dataset and shows superior performance to relevant baselines. The experiments on the medical imaging datasets introduce synthetic noise to the segmentations while the cityscapes task uses coarse-grained segmentations to predict the fine-grained ones.", "summary_of_the_review": "The paper introduces a novel method for training with noisy segmentation labels but might be biased in its evaluation and comparison to relevant baseline methods. Furthermore, I am not sure whether the assumptions made for the noise-model are as widely applicable as claimed.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The method appears to be novel and produces competitive or superior performance on the shown experiments.\n- The method is an interesting step towards spatially-aware modelling of segmentation noise.\n\nWeaknesses:\n- The experiments seem like they might be biased. The medical imaging experiments only use the proposed noise model to add synthetic label noise. How would the model perform with a mismatch in label noise? One potential avenue for studying this would be to use a dataset such as LIDC-IDRI that comes with multiple annotators and define the ground truth as a majority voting of the individual labels and select a single rater as the noisy annotations.\n- The method implicitly assumes contiguous regions without holes to be segmented. How would this method perform with potential holes within the segmentation masks?\n\n\nMisc:\n- In the related work you mention that superpixel based approaches are not capable of addressing the challenges of noisy segmentation labels. Could you elaborate on this?\n- I am curious how this method would compare to weak segmentation methods that only use bounding boxes etc as labels. One could argue that the problems that are solved in both cases are very similar.\n- I am curious about the relationship between this method and methods that aim to model spatially aware uncertainty of the segmentation predictions (e.g. [1]). Would spatial awareness in the modelling be sufficient to then calibrate the predictions based on the estimated bias of the annotations?\n- p3: \"net-work learning\" -> \"network learning\"; \"Section 3.1, We aim\" -> \"we aim\"; the penultimate paragraph could use a bit of editing\n- p4: \"N_S\" is only introduced in definition 2 but used earlier\n- Section 3.1 could be improved by adding a bit more intuition about the equations.\n- Notation: I generally would prefer reading $\\mathbb{E}[Y]$ than $\\mathbb{E}Y$\n- p5: exits -> exists ?\n- page 8 mentions sparse labels - what do you mean by that?\n- How many seeds do you use for the standard deviations?\n\n[1] Monteiro, Miguel, et al. \"Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty.\" Advances in Neural Information Processing Systems 33 (2020): 12756-12767.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster