Cut out the annotator, keep the cutout: better segmentation with weak supervision
Authors: Sarah Hooper
Michael Wornow
Ying Hang Seah
Peter Kellman
Hui Xue
Frederic Sala
Curtis Langlotz
Christopher Re
Venue: ICLR 2021 Poster (2021)
Keywords: Weak supervision
segmentation
CNN
latent variable
medical imaging

Abstract: Constructing large, labeled training datasets for segmentation models is an expensive and labor-intensive process. This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS). Such techniques, however, have limitations when applied to image segmentation---FSL methods often produce noisy results and are strongly dependent on which few datapoints are labeled, while WS models struggle to fully exploit rich image information. We propose a framework that fuses FSL and WS for segmentation tasks, enabling users to train high-performing segmentation networks with very few hand-labeled training points. We use FSL models as weak sources in a WS framework, requiring a very small set of reference labeled images, and introduce a new WS model that focuses on key areas---areas with contention among noisy labels---of the image to fuse these weak sources. Empirically, we evaluate our proposed approach over seven well-motivated segmentation tasks. We show that our methods can achieve within 1.4 Dice points compared to fully supervised networks while only requiring five hand-labeled training points. Compared to existing FSL methods, our approach improves performance by a mean 3.6 Dice points over the next-best method. 

Published as a conference paper at ICLR 2021
CUT OUT THE ANNOTATOR, KEEP THE CUTOUT: BETTER SEGMENTATION WITH WEAK SUPERVISION
Sarah M. Hooper'; Michael Wornow", Ying Hang Seah",
Peter Kellman', Hui Xue', Frederic Sala², Curtis Langlotz', Christopher Ré² 'Department of Electrical Engineering. Stanford University, Stanford CA, USA "Department of Computer Science, Stanford University, Stanford CA, USA National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda MD, USA DDepartment of Radiology, Stanford University, Stanford CA,, USA
ABSTRACT
Constructing large, labeled training datasets for segmentation models is an expen- sive and labor-intensive process. This is common challenge in machine learning. addressed by methods that require few or no labeled data points such as few-shot learning (FSI.) and beakkl-suupevisse learning (WS). Such techniques, however, have limitations when applied to image segmentation- FSI. methods often pro- duce noisy results and are strongly dependent on which few datapoints are labeled, while WS models struggle to fully exploit rich image information. We propose a framework that fuses FSL and WS for segmentation tasks, enabling users to train tigh-performin segmentation networks with very few hand-labeled training points. We use FSL models as weak sources in a WS framework, requiring a very small of reference labeled images, and introduce new WS model that focuses on key areas areas with contention among noisy labels-of the image to fuse these weak sources. Empirically, we evaluate our proposed approach over seven well-motivated segmentation tasks. We show that our methods can achieve within 1.4 Dice points compared to fully supervised networks while only requiring five hand-labeled training points. Compared to existing FSL methods, our approach improves performance by mean 3.6 Dice points over the next- best method.
INTRODUCTION
Automated image segmentation has seen rapid improvements with recent developments in deep learning (Li et al. 2018 Chen al. 2017 Milletari et al. 2016). Convolutional neural networks (CNNs) achieve high segmentation performance but can require large, labeled training datasets. Acquiring training labels is laborious and slow, particularly for medical images where expert seg- mentation is often required in 3- or 4-dimensions. While large datasets and pre-trained networks exist for natural images, medical image segmentation is more targeted task, typically requiring new training sets for every imaging modalíty, scanner type, anatomical structure, and patient popu- lation. Such difficulties abound, but the significant impact of improved medical image segmentation motivates tackling these challenges (Hesamian et al. 2019).
Many few-shot learning (FSL) approaches have been proposed mitigate these difficulties by train- ing networks using only a few labeled examples. For example, data augmentation can reduce the needed amount of labeled data by introducing additional variation into small, labeled train- ing sets through operations such affine transforms or learned transformations (e.g. deformations learned by GANs) (Zhao et al. 2019. FaaoonRRose et al., 2018 Çiçek et al.) 2016). Many semi- supervised approaches aim to learn useful representation from unlabeled data then fine-tune few manually-annotan
images (Chen et al., 2020 Bai et al. 2019 Chen et al. 2019 Chaitanya ct al. 2020a). Finally, other approaches aim to transter know ledge learned from segmenting one class in order to segment a previously unseen class (Shaban et al. 2017: Rakelly et al. 2018 Roy et al.] 2020; Ouyang et al. 2020). However, these FSL approaches have limitations. For example, effective data augmentation often requires model- and task-specific transformations that are difficult
FFrr correspondence, please contact emhooper@st.nnfrr.... @ttanforo.edd

---

Published as a conference paper at ICLR 2021
Step Label data with noley labeling functions (UFs]
Challenge 2 visuolied
Learn a madel to combine Lf outputs
Train a CNN using the rggregated labels
+
Accumcy
Challenge Injecting knowledge
Challenge Modeling Accuracios
Unlobeled data
[o]
(b)
Figure 1: Left: Proposed pipeline (blue) and challenges (red). Multiple LFs noisily label data; label model (LM) then aggregates the LF labels into weak labels that can be used to train CNN. Two key challenges: how to build LFs that inject knowledge and how to correctly model LF accuracies. Right: Second challenge- issues using global accuracy as a metric in the LM: the accuracies of segmentation masks that clearly correct (b, top) are similar to those that clearly incorrect (b, bottom). Additionally, segmentation accuracies can be arbitrarily changed by cropping (c).
to generalize to all tasks, and transferring knowledge from segmentation target to another relies on the segmentations of other classes being present in training datasets. More generally, bringing all of the known domain expertise structural information to bear is prohibitive.
Weak supervision (WS) a different approach to training without manual labels that takes advantage of multiple weak labeling sources (Mintz al.) 2009 Craven & Kumlien 1999, Takamatsu al.) 2012 Gupta & Manning, 2014] Ratner et al.) 2019). In many WS workflows, labeling functions (LFs) are crafted encode the pecision-making processes of experts into small functions that pro- gramatically label training data. LFs can drastically reduce labeling time, but the resulting labels are often noisy and conflicting. The driving idea behind WS is that the accuracies of and correlations between LFs can be learned without access to ground truth labels, then used to aggregate the LF outputs. This WS workflow enables rapid construction of training sets without manual annotation.
Due to the widespread success of weak supervision methods (Dunnmon et al. 2020 [hhng et al.) 2020 Bach et al. 2019), weak supervision is an appealing approach for generating labeled segmen- tation training sets without relying on hand-labeled data. However, existing WS frameworks do not readily extend to the segmentation setting for the following two reasons:
Injecting knowledge. It is unclear how users construct multiple segmentation LFs that inject knowledge for a segmentation task at hand-and and are suitable for any image. Modeling LF accuracies. We need to fuse the LFs, but the probabilistic graphical mod- els (PGMs) of existing WS frameworks rely on estimating the accuracies for each labeling function, which is problematic metric for segmentation: it ences at segmentation borders nor robust to arbitrary image crops (Figure [ right).
sensitive small differ-
We propose best-of-both-worrf approach that fuses FSL. and WS for segmentation tasks. Specif- ically, we use FSL models as LFs, obviating the need to craft expertise-injecting LFs by hand. To resolve the difficulty tracking accuracies per LF, which are too coarse of a metric for segmenta- tion, we proposing a new WS PGM. Our PGM introduces conditional terms that "focus" on areas of contention, where LFs disagree, more finely tracking the LF performance in such areas.
We validate our approach seven medical image segmentation tasks. We show that our method can achieve within 1.4 Dice points of fully supervised network using only five labeled images. We compare our proposed method to five other few-shot segmentation methods, and find that our method exceeds the nest-best by a mean 3.6 Dice points. Finally, we analyze how different FSL methods scale with additional unlabeled training data and compare PGM to existing WS models.
There are two primary contributions in this work. First, we present a weak supervision pipeline tailored to the segmentation use case which utilizes small number of labeled images and many unlabeled images to train a high-performing segmentation network. Second, we develop a novel PGM to aggregate noisy segmentation masks and show that the parameters of this model can be learned without access to ground truth labels. We show that our new approach to few-shot image segmentation outperforms many competitive baselines.
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
Data augmentation Data augmentation has been shown to enable training on few labeled exam- ples, particularly medical imaging. These augmentations range from simple operations such as affine transforms, elastic transforms, and contrast jitter (Çiçek et al.) 2016 Cireşan et al. (01T) to more complex learned operations, such as the use of GANs learn intensity and spatial deformation transforms to generate synthetic medical images (Chaitanya et al. 2020b).
Transferring knowledge A popular FSL approach enables prediction on previously unseen classes by leveraging representations learned from prior experience (Snell al. 2017 Sung al.)
2018). Applications to segmentation include Rakelly et al. 20: Shaban et al. (01 (177; Dong &
Xing 2018). For example, PANet is proposed approach for few-shot semantic segmentation that utilizes representations for each semantic class from the support set, then labels query images by
propagating labels from the support set in embedding space (Wang al.) 2019). Few-shot segmen- tation was extended specifically to the medical imaging case by Roy et al. (2020), where "squeeze and excite" blocks are used to learn class representations without pretrained networks. However, these approaches rely the availability of labels of other classes during training.
Self- and semi-supervised learning Self- and semi-superviisd methods have been proposed for image segmentation without relying the availability of other classes during training (Chen al., 2020 Bai et al. 2019; Chen et al. 2019). Instead, semi-supervised segmentation methods often pretrain on large unlabeled datasets to learn useful representations without any annotations. For example, contrastive learning has been proposed to pretrain CNNs, which are then fine-tuned using small number of manually segmented images (Chaitanya et al.) 2020ap. Additionally, superpixel segmentations have been proposed to self-supervise networks, which can be used to segment test images without fine tuning (Ouyang et al. 2020).
In the Appendix, we also discuss conditional random fields and using coarse labels (e.g., bounding boxes, scribbles, and image-level labels) train segmentation models.
SEGMENTATION WITH FSL AND WS FUSION
We first give background on WS and state the problem setting. Then, we follow with details of the primary challenges and solutions in each stage of method. Specifically, we diseuss LFs and new PGM that fuses FSL with WS. We end with summary of our proposed workflow.
3.1 WEAK SUPERVISION BACKGROUND
We build approach off of principles from WS. We note that WS is overloaded term in lit- erature. In this work, we refer to WS as the class of approaches in which multiple noisy labeling sources are aggregated into probabilistic training labels without access to ground truth labels (Ratner ct al. 2019).
The goal of WS is to best utilize multiple sources of noisy signal (i.e. LFs) label training data without relying on manual annotation. Each LF takes as input an unlabeled training point and outputs vote for the label of that training point. These LFs can replace manual training data annotation, however LF votes can be imperfect and conflict with one another. A latent variable model (referred to as label model) can be used to model these observed LFs and their relationship to the latent ground truth label. In Figure 2a. show simple model where five LFs each vote the label for a single unobserved ground truth label. Importantly, the parameters of this model need to be estimated without access to ground truth annotation. The learned label model can be then used to produce probabilistic training labels to train a downstream network, referred to as an end model.
Building off of this past work, weak supervision pipeline proceeds in three steps: (1) noisy labels are generated by multiple LFs, (2) the noisy labels are aggregated into weak labels by a probabilistic graphical model (PGM), and (3) the weak labels used to train segmentation model (Figure
3

---

Published as a conference paper at ICLR 2021
Figure 2: Example latent variable models. (a) A simple PGM with one latent variable Y and several observed labeling functions X² (b) Conventional segmentation grid. (c) The PGM used in this work, where all nodes in grid of latent variables weakly labeled by labeling functions.
3.2 PROBLEM SETTING
We are given a training set of n images, X [X,, Xn]. where X; E RXXXXXXT is a tensor of P pixels. J. K, and T represent the dimensions of image X,, where K > 1 for volumet- segmentation mask
ric images and T >> for time series images. For each image X; there Y; (c)/xXxXXXX where {c} is the set of class labels, yielding the set Y [Y Y,.,.. Yn] of all unobserved ground truth segmentation masks. We have access to m labeling functions that provide noisy segmentation masks X (c)/x/xXXX for each X,, where
{1, 21,2..., m]. From these observed sources, we seek generate the set of weak labels & estimate Y. & provides labels for a large dataset, and we can use these to train any end model (e.g., a segmentation CNN).
We tackle the binary segmentation case, {c} (-1, 1): simple multiclass extensions possible.
Model Structure The (unobserved) ground truth mask Y and the (observed) sources À are jointly modeled by an undirected graphical model (Koller Friedman, 2009; Wainwright & Jordan 2008 with graph G (V, E). Since Y is unobserved, this is satent-variable model. We use conventional structures for each component and combine them in a natural way:
In standard weak supervision approach, the weak sources connected to ground- truth label Y (Fig. 2h),
The segmentation mask Y is multi-dimeensinal grid over the pixels (Fig- 2),
Our full model has one copy of the WS graph for each pixel in the grid (Fig. 2
Our goal construct effective LFs and complete the model specification above by defining the potentials associated with the edges in E and the corresponding parameters.
3.3 INJECTING KNOWLEDGE WITH FSL-BASED LABELING FUNCTIONS
In Step I of our workflow (Figure D. we use LFs to generate multiple noisy segmentation masks for each unlabeled image. These LFs transfer the decision-making processes about labeling data into functions that can be run programmatically over unlabeled training data.
Challenge LF development typically involves writing short programs expressing heuristics (Rat-) ner et al., 2018) or external knowledge base lookups (Mintz al. 2009 Takamatsu et al. (012)) However, even with computer vision expertise, it is difficult for new user to write segmentation heuristics as LFs for each new task hand, is required of users from existing WS tools.
Solution We propose FSL models unified way to inject labeling knowledge into LFs. In this work, we use one-shot learning: we train a CNN from a single labeled data point for each LE. However, note that any few-shot model could be used labeling function. Fusing FSL and WS allows us to build upon the strengths of each:
From WS, we build from the idea that we can track the noise in multiple labeling sources to aggregate labels. We improve upon existing WS tools by showing that fhw- networks-
4

---

Published as a conference paper at ICLR 2021
which obviate the need for users to write their own LFs, as is required by existing WS frameworks are flexible and strong LFs in a WS segmentation pipeline.
From FSL, we take ideas for how to train networks with very few labeled images. We improve upon these methods by tracking the noise in each FSL model with WS.
Our FSI--based LFss depart from the dopain-expert LF development approach, which is often re- peated for each task. Instead, the same process followed for all tasks explored here: five images annotated by hand, each of which are used to train one-shot model. The one-shot models func- tion as LFs that weakly label the rest of the training set. In Section A.2 of the Appendix, we discuss other segmentation LFs considered.
3.4 MODELING ACCURACIES WITH CONTENTION-BASED PROBABILISTIC MODEL
In Step 2 of our workflow (Figure D. aggregate the noisy segmentation masks produced by the LFs into probabilistic training labels.
Challenge To aggregate the outputs of the LFs, we need define the graph structure G (given in FigureZe) and the potentialspparaeetess associated with the edges in G. However, it is not obvious how to do so. On one extreme, separate parameters for each edge in the model in Figure ? easily results in millions of parameters to be learned. At the other extreme, treating each pixel as indepen- dent of all other pixels (as in separate copies Figure (2) results in the problems summarized in Figure [ (right): the parameters tend to be equivalent, since all LF accuracies are high and similar, and can be changed by arbitrary crops.
Solution Our approach hits sweet spot. It exploits a notion of LF contention, trading off between the naive model and the highly-complex model that has no parameter-tying.
Model Potentials, Parameters, and Contention
The latent variable model in Figure 2a expresses the relationship between the sources and unob- served ground truth label. The potentials are typically those from a binary Ising model (Bach al.) 2017
Ratner et al. 2019 Varma et al., 2019 Fu et al. 2020). The density function is defined as,
Al,,, Am)
exp OOyY+ Σ.Ar)
(1)
where By E IR and 0, E R. Intuitively, the 0, parameters control the accuracies of the sources. If 0, = 0, then > is independent of and thus only output random guess of Y. As A increases, A; and Y vote together more often, and so X; is more accurate. For this reason, is crucial to estimate the parameters of this model; naively combining the noisy labels will over-weight the inaccurate sources.
In our segmentation structure (Fig. (p), there many copies of this model. Using unique parame- ters for each copy yields an intractably complex model. Tying parameters naively (i.c., each LF has the same accuracy across all pixels) is too coarse-even poor LF will typically be very accurate across the entire image. To tackle this, we introduce a notion of LF contention into our density function. Specifically, we use indicator variables that depend on LFs agreeing or disagreeing on a given pixel. Intuitively, this will allow the model to learn a certain set of parameters when the LFs are all agreeing (e.g. image background) vs. when the LFs have some disagreement (e.g. seg- mentation border). By tying parameters based on LF agreements, this distribution allots additional model parameters to areas of contention without introducing too many free parameters to model the grid-structured PGM. Moreover, this approach enables differential model parameters to be learned even when global LF accuracies are highly similar, and is robust to arbitrary image crops.
We refer to of LF agreement conditions over LF subset D Cp- CD exhaustively enumerates each possible combination of LF agreements and disagreements, C, For example, the Cp for D {>1, As) is given by
Cp - CC Cz, Ci)
C = [A1 + X2, A
Aa). C2 = [} * X2, 21 =1 do
5

---

Published as a conference paper at ICLR 2021
C3 = [A1 A2 + A2 As), (A1 =2, d2
technical detail we exploit for learning: we always leave two LFs out of the contention to ensure sufficient conditional independence for parameter recovery, as we describe next. The set of these left-out LFs is /; there are m = 5 LFs, then {d4, >s}.
Formally, we define the density of the contention-baset model as
p(Y,, .."",,""
AT")
ICI [(C
P
Z exp
Σ p=1
(0y, + OxEAP +
E
Ory, VPY!)
(2)
REDUI
(p.))E
Weak supervision
Segmentation
where each canonical parameter 0g E IR. In practice we only sum over a single element of D, which an equivalent but minimal distribution. enabling backwards mapping as described below.
Learning & Inference
Given observed values of LFs over set of images, wish learn the parameters from (2) Critically, this must be done without relying on ground truth segmentation masks-so that only the variables, and not Y, are observed.
We use an extension of Fu et al. (2020)'s mf [[A,A,] = EEA,Y] The left-hand side can be estimated (as we observe the LFs); system with three such equations, for the pairs (i.j), (i. k), (j, k). determines E[AA ] up to sign. These terms are the mean parameters of the model, and under mild technical conditions, they determine the canonical parameters A (Wainwright & Jordan 2008). Our extension incorporates the contention term into the decomposition. The full details are the Appendix A.3
approach. For the model (
After solving for the label model parameters, our goal is to compute the marginal probability for each pixel Y given the observed values of the labeling functions for pixels our graph. Various methods have been proposed for performing inference over large and cyclical graphical model. Here, we approximate the exact conditional using a common graph-cut algorithm. We learn the parameters and compute the unary potentials at each pixel, treating each pixel independent at this stage. The max-flow/min-cut algorithm proposed by [Boykov & Kolmogorov (2004) is used to efficiently compute the final segmentation mask considering pairwise interaction between all neighboring pixels. However, we note that many approximate inference algorithms could be used as well, including approximate belief propagation algorithms that would return probabilistic labels.
3.5 SUMMARY WORKFLOW
Taking the solutions developed in Sections 3.3 and 3.4 we put the pieces together and describe the full workflow of our proposed approach (Figure
Step 1. Each one of the five manually annotated images is used to train a one-shot CNN.
With five manually annotated images, there are five one-shot CNNs. Each of these one-shot CNNs is labeling function which labels all unlabeled images in the training set.
Step 2. PGM used to aggregate the five noisy segmentation masks for each image in the unlabeled training set. The parameters of this PGM are learned for each dataset without using ground truth segmentation masks.
Step 3. CNN trained using the estimated segmentation masks output from the PGM, then fine-tuned with the few hand-labeled images.
4 EXPERIMENTS
The primary goal of our evaluation is to show that proposed weak supervision method can achieve high performance across range of medical image segmentation tasks (Section 4.2).. Secondly, we aim to more deeply understand the benefits of the proposed method compared other methods (Section 4.3
We describe our experimental setup in Section 4.1

---

Published as a conference paper at ICLR 2021
Step
Step2
Step 3
Inpul: S labeled images; many images
Output: trained
PGM
Uniabored
Input step
Input oggropeted each uniobeled image
unlobeled image
Figure 3: Details of proposed workflow. In Step I, each manually-annotate image is used to train a one-shot model, which turn produces noisy label for all unlabeled training images. In Step the outputs from the FSL LFs are aggregated via PGM, the parameters of which are learned without access to any ground truth labels. Step the PGM outputs are used train an end model.
4.1 EXPERIMENTAL SETUP
Datasets We select public segmentation datasets to evaluate our proposed method. We chose datasets to include varied imaging modalities, body regions, and structures.
ACDC. 100 volumetric short-axis cardiac magnetic resonance (CMR) cine images avail- able for download, totaling 951 2D axial slices. The right ventricle, the left ventricle endo- cardium, and the myocardium segmentations were provided (Bernard et al., 2018).
Abdominal CT. 43 contrast-enhance abdominal computed tomography (ACT) scans, to- taling 10,235 2D axial slices. We use the segmentations provided for the spleen, esophagus, and liver (Gibson al. 2018 Roth al., 2016 2015) Clark al., (003)).
BRATS. 210 head magnetic resonance (HMR) scans, totaling 31,775 axial slices. We use the flair HMR sequence and segment any region positive for tumor (Menze et 2014
Bakas et al. 2017 2018).
Baselines We compare proposed approach five other few-shot segmentation methods. We select the baselines that have similar inputs (a few labeled images and many unlabeled images) and outputs (a trained segmentation neural network) and have published implementations by the authors. In each case, we use the same unlabeled and in our proposed approach to ensure the training datasets comparable.
proposed approach, but use distinct method,
five labeled images
Data augmentation. We compare against data augmentation method that trains gen- erative adversarial networks (GANs) using unlabeled images, then transforms the labeled images using the trained GANs to generate larger labeled training set ("GAN DA") (Chai-
tanya et al., 2020b).
Knowledge transfer. We compare against PANet (Wang al. 2019 FSL approach semantic segmentation developed for natural images. We also compare against Squeeze and Excite ("S&E"", a few-shot approach developed for medical images (Roy al.) 2020). Semi- and self-supervised learning (SSL). We compare against a semi-supervised method that pretrains using a contrastive loss over many unlabeled images, then fine-tunes on few manually-annotate images ("CL"" (Chaitanya et al.) (020a). We also compare against SSL-ALPNet, which utilizes a self-supervised superpixel segmentation task then uses the learned representations to segment new classes without fine-tuning (Ouyang et al., 2020). Traditional supervision. We train a model using only the five labeled reference images the lower bound of expected performance ("FS-5"".. We also train fully supervised network using all ground truth segmentations of all training data as the upper bound of expected performance ("FS-all").
7

---

Published as a conference paper at ICLR 2021
Table 1: Performance of proposed method and baselines.
Cardiac MR Myo.
BRATS
Method
Endo.
RV
Liver
Spleen
Esophagus
Tumor
FS-5
(1.2)
55.5 (2.4)
(8.7)
92.0 (0.7)
92.7 (0.3)
48.2
77.1 (0.5)
GAN DA
90.8 (2.7)
76.3 (0.4)
(0.4)
86.9 (1.9)
(5.0
54.9 (0.8)
73.9 (4.1)
PANct
(2.0
(8.2)
42.3
(8.7
23.0
217.3)
N/A
S&E
(6.9
(2.3)
(1.7
(3.2)
N/A
SSL-ALPNet
(3.0(1.4)
(1.3)
70.7 (1.7)
(0.3)
(0.7
36.6
(1.4))))
J0.5((1.0)
(1.11
81.1 (2.0)
91.8 (0.3)
87.8 (2.8)
(3.1)
78.4 (0.8)
Proposed
94.7 (0.5)
81.0 (1.2)
87.2 (3.2)
93.2
93.1 (9.4)
69.2
77.4 (0.7)
FS-all
96.1 (0.3)
87.0 (0.5)
93.6 (0.1)
96.1 (0.2)
94.5 (0.3)
75.6 (2.1)
83.5 (0.5)
Figure 4: Example results. The left of each pair is the ground truth, the right of each pair produced by the proposed method. Left: orange is endocardium, pink is myocardium, yellow is right ventricle. Right: orange is spleen, pink liver, yellow is esophagus.
Training and Evaluation Each dataset is split into 60% training, 20% validation, and 20% testing- Image preprocessing is described in the Appendix A.4 For the LFs, FS-5, FS-all, and our proposed approach, we train a 2D U-Net (Ronneberger et al. 2015) ) to minimize the binary cross entropy loss using the Adam optimizer with learning rate set to 1e-3. Batch size set 16. Random affine transforms, color jitter, and elastic transforms were used during training for all models except FS-5. We note that the performance of FS-all will be lower than that reported in public leaderboards due to our simple end model. Because we aim to show how well our proposed method utilizes unlaheled data and a small of labeled data, we keep the end model simple to be comparable baselines. More complex networks could easily be substituted into Step 3.
The Dice coefficient, Dice((Y, Y)) trained with three different random seeds. The mean (standard deviation) Dice scores are reported.
2| is used to evaluate networks. Each network was |
4.2
PERFORMANCE OF PROPOSED METHOD
4.2.
COMPARISON TO BASELINES
The Dice scores of our proposed approach and baselines are shown for the seven segmentation tasks in Table [ visualizations of the segmentations are in Figure E and in the Appendix Figure A. Despite using only five labeled images, we obtain near fully supervised performance on the endo- cardium, liver, and spleen, achieving average within 1.9 points of the fully supervised network. On the more difficult segmentation tasks, we still see boost compared to existing approaches, though there is larger gap between FS and WS performance. This is to be expected, since the segmentation of smaller structures more difficult. Still, we see strong agreement in the shape and location of the WS segmentations compared to the ground truth segmentations.
4.2.2 INFLUENCE OF SELECTED LABELED IMAGES
We evaluate how our results change depending on the five manually-annotatee images. The original set of manually-annotall
images were selected randomly. We select three additional sets of refer- ence images and re-run proposed method; all sets are mutually exclusive. The results from each new set (run with the same random seed) are shown in Table 2 In general, we see good agreement across different randomly seleected labeled images. For all datasets except the right ventricle and the esophagus, we see standard deviations less than 1.5 Dice points. Selecting images to label from unlabeled dataset improve end model results would be interesting direction for future work.
8

---

Published as a conference paper at ICLR 2021
Table 2: Performance of proposed method with different sets of labeled images.
Cardiac MR
Ab. CT
BRAIS
Labeled image act
Endo.
Myo.
RV
Liver
Spleen
Esophagus
Tumor
Original set
94.9
79.7
83.9
93.7
92.9
70.4
77.4
Set
93.4
81.4
89.9
95.6
93.1
73.2
75.4
Set 2
94.9
81.8
89.5
94.5
91.1
51.6
76.5
91.7
66.1
75.3
0.70
0.50
Propoood INte
2045
Velertyy Uhie Proposed
0.40
0355
0.30
0.2
0A
Figure 5: Comparisons for different amounts of unlabeled data (left) and image crops (right).
4.3 TRADEOFFS WITH OTHER APPROACHES
Performance scaling with amount of unlabeled data Since unlabeled data is much easier to collect than mamually-annotatee data, FSL approaches that are able to effectively utilize unlabeled data may have advantage when deployed real-world applications. In Figure 5 (left), we show the performance of our proposed approach, the highest performing knowledge transfer approach (PANet), the data augmentation network (GAN-DA), and the highest performing self- or semi- supervised network (CL) as more unlabeled data is provided during training for the CMR right ventricle task. We observe that the performance separation between WS and other approaches is larger with more unlabeled data. In the low data regime, WS, CL, and GAN-DA perform similarly.
Label model performance To show the advantages of the proposed segmentation LM versus existing WS models, we conduct experiment with synthetic datasets. We generate dataset according to Equation 2 and pad this dataset with negatively labeled examples. This process mimics cropping image around the ROI. We show the performance of proposed LMM, majority vote (MV), and leading existing WS framework (Flying Squid, Fu et al. (2020) in Figure (rigut)((righh))) (right). We see our proposed model is unaffected by background cropping (or equivalently, LF disagreement rate) while the existing WS tool has varying performance as a function of arbitrary crops. We report the Dice score of MV, the proposed LM, and Flying Squid in the Appendix[A.4 Finally, Appendix A.4 we also include an experiment on the generalization performance of the FSL models.
5 CONCLUSION
Automated medical image segmentation is difficult problem. There are numerous imaging modal- ities, views, and structures to segment and each application demands high performance, leading to extensive labeling costs from experts. We present general approach to training segmentation networks with limited hand labels, fusing FSL and WS to build upon the benefits of each. This approach is flexible, capable of utilizing various forms of noisy signal to weakly label training data, which can in turn be used to train any downstream model. Our method compatible with future improvements in segmentation methods: and more complex segmentation networks or loss functions can be used in Step 3. For example, unsupervised pre-training could be used to initialize the one-shot networks used LFs, such the methods proposed in Chaitanya et al. (2020a). In fact, any of the baselines we compare against in this work could be used as LFs. Finally, we have shown that the proposed approach to training seg- mentation networks with few labels outperforms many competitive baselines on range of different imaging modalities and anatomical structures.
few-shot approaches
be used as LFs in Step 1,

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
Sarah Hooper supported by the Fannie and John Hertz Foundation, the National Science Foun- dation Graduate Research Fellowship under Grant No. DGE- and as a Texas Instruments Fellow under the Stanford Graduate Fellowship in Science and Engineering. We gratefully acknowl- edge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCFI763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML): ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qual- comm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss Re, Total, the HAI-AWS Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. The Mo- bilize Center is a Biomedical Technology Resource Center, funded by the NIH National Institute of Biomedical Imaging and Bioengineering through Grant P41EB027060. The U.S. Govermmnnt is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorse- ments, either expressed or implied, of NIH, ONR, or the U.S. Government.
REFERENCES
Stephen H. Bach, Bryan He, Alexander J. Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017), Sydney, Australia, 2017.
Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen. Alex Ratner, Braden Hancock, Houman Alborzi, et al. Snorkel drybell: A case study in de- ploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference on Management of Data, pp. 362-375, 2019.
Wenjia Bai, Chen Chen, Giacomo Tarroni, Jimming Duan, Florian Guitton, Steffen E Petersen, Yike Guo, Paul M Matthews, and Daniel Rueckert. Self-supervised learning for cardiac mr image segmentation by anatomical position prediction. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 541-549. Springer, 2019.
Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 4: 1701 17, 2017.
Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv: /8/ 1.02629, 2018
Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, al. Deep learning techniques for automatic mri cardiac multi-strructures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging, 37(11):2514-2522 2018
Yuri Boykov and Vladimir Kolmogorov. An experimental comparison of min-cut/max-flot flow algo- rithms for energy minimization in vision. IEEE transactions on pattern analysis and machine intelligence, 26(9): 124-1137, 2004.
Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence, 23(11):122--133, 2001.
Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. arXiv preprint arXiv:2006. 105/ 2020a.
10

---

Published as a conference paper at ICLR 2021
Krishna Chaitanya, Neerav Karani, Christian F Baumgartner, Anton Becker, Olivio Donati, and En- der Konukoglu. Sumi-super task-driven data augmentation for medical image segmentation. arXiv preprint arXiv:2007.053663 2020b.
Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueck- ert. Self-supervised learning for medical image analysis using image context restoration. Medical image analysis, 58: 10:199,,, 2019.
Liang-Chieh -highh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 1XXiv:7766005877 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
Özgün Çişek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u- net: learning dense volumetric segmentation from sparse annotation. International conference medical image computing and computer-assister intervention, pp. 424 432. Springer, 2016.
Dan C Cireşan, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High- performance neural networks for visual object classification. arXiv preprint arXiv:1102.0/83, 2011.
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia): maintaining and operating a public information repository. Journal of digital imaging, 26(6): 1045-1057, 2013.
M. Craven and J. Kumlien. Constructing biological knowledge bases by extracting information from text sources. In International Conference on Inielligent Systems for Molecular Biology (ISMB), 1999.
Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 3, 2018.
Jared Dunnmon, Alexander Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew Lungren, Daniel I. Rubin, et al. Cross-modal data programming enables rapid medical machine learning. Putterns, pp. 100019, 2020.
Zach Raton-Rosen, Felix Bragman, Sebastien Ourselin, and M Jorge Cardoso. Improving data augmentation for medical image segmentation. 2018.
Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. arXiv preprint 10X2: 2020.
Eli Gibson, Francesco Giganti, Yipeng Hu, Ester Bonmati, Steve Bandula, Kurinchi Gurusamy, Brian Davidson, Stephen P Pereira, Matthew Clarkson, and Dean C Barratt. Automatic multi- organ segmentation on abdominal et with dense v-networks. IEEE transactions on medical imag- ing. 37(8):1822-1834, 2018.
Sonal Gupta and Christopher D Manning. Improved pattern learning for bootstrapped entity extrac- tion. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pp. 108, 2014.
Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul Kennedy. Deep learning tech- niques for medical image segmentation; Achievements and challenges. Journal of digital imag- ing, 32(4):582-59) 2019.
A. Emre Kavur, N. Sinem Gezer, Mustafa Bany, Pierre-Hent Conze, Vladimir Groza, Due Duy Pham, Soumick Chatterjee, Philipp Erust, Savaş Özkan, Bora Baydar, Dmitry Lachinov, Shuo Han, Josef Pauli, Fabian Isensee, Matthias Perkonigg, Rachana Sathish, Ronnie Rajan, Sinem Aslan, Debdoot Sheet, Gurbandurdy Dovletov, Oliver Speck, Andreas Nümberger, Klaus H.
11

---

Published as a conference paper at ICLR 2021
Maier-Hein, Gözde Bozdağı Akar, Gözde Ünal, Oguz Dicle, and M. Alper Selver. CHAOS Challenge Combined (CT-MR) Healthy Abdominal Organ Segmentation, January 2020a. URL
atttps://arae org/ abs/ 2ttp:/201..0000
A. Emre Kavur, Naciye Sinem Gezer, Mustafa Banı, Yusuf Şahin, Savaş Özkan, Bora Baydar, Ulas Yüksel, Çağlar Kılıkęer, Sahin Olut, Gözde Bozdağı Akar, Gözde Unal, Oğuz Dicle, and semi-automatic and deep learning based automatic methods for liver segmentation in living liver transplant donors. Diagnostic and Interventional Radiology, 26: 11-21, January 2020b. doi: 10.5152/dir.2119.U URthttss:://dee ott.///....ro11111///55.115
M. Alper Selver. Comparison
0019.19022
Ali Emre Kavur, M. Alper Selver, Oguz Dicle, Mustafa Barış, and N. Sinem Gezer. CHAOS Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, April 2019. URL
https://doi. org/10.......o....o.o...o....... 2281/zenodo. 3362844
Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 876-885 2017.
Pushmeet Kohli, M Pawan Kumar, and Philip HS Torr. P3 & beyond: Solving energies with higher order cliques. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007.
Pushmeet Kohli, Philip HS Torr, et al. Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 2233)302-3244 2009.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. In Advances in neural information processing systems, pp. 109-117, 2011.
Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi-Wing Fu, and Pheng-Ann Heng. H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE transac- tions on medical imaging. 37(12):2663-2677 2018.
Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Salpathy-Crammes Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34 (10):1993-200 2024, 2014.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-571. IEEE, 2016.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2- Volume 2, pp. 1003-1011 Association for Computational Linguistics, 2009.
Cheng Ouyang, Carlo Biffi, Chen Chen, Turkay Kart, Huaqi Qiu, and Daniel Rueckert. Self- supervision with superpixels: Training few-shot medical image segmentation without annotation. arXiv preprint arXiv:2007 009886, 2020.
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A Efros, and Sergey Levine Few-shot seg- mentation propagation with guided networks. arXiv preprint arXiv: 180.007800 2018.
A. J. Ratner, Hancock, Dunnmon, F. Sala, S. Pandey, and C. Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, 2019.
A.J. Ratner, S.H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré. Snorkel: Rapid training data creation with weak supervision. In VLDBB 2018.
12

---

Published as a conference paper at ICLR 2021
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer. 2015.
Holger Roth, Amal Farag, Evrim Turkbey, Le Lu, Jiamin Liu, and Ronald Summers. Data from pancreas-ct. 2016.
Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan: Multi-level deep convolutional networks for automated pancreas segmen- tation. In International conference medical image computing and computer-assistd interven- lion, pp. 556-564. Springer, 2015.
Abhijit Guha Roy, Shayan Siddiqui, Sebastian Pölsterl, Nassir Navab, and Christian Wachinger. "squeeze & excite' guided few-shot segmentation of volumetric images. Medical image analysis, 59:101587, 2020.
Tomas Sakinis. Fausto Milletari, Holger Roth, Panagiotis Korfiatis, Petro Kostandy, Kenneth Philbrick, Zeynettin Akkus, Ziyue Xu, Daguang Xu, and Bradley Erickson. Interactive segmentation of medical images through fully convolutional neural networks. arXiv preprint arXiv:1903.08205, 2019.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:/709.03:: 700.003499 2017.
Ying Sheng, Nguyen Ha Vo, James B Wendt, Sandeep Tata, and Marc Najork. Migrating privacy- safe information extraction system to a software 2.0 design. 2020.
Jamie Shotton. John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for image under- standing: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International journal of computer vision, 81(1):2-23, 2009.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
S. Takamatsu, I. Sato, and H. Nakagawa. Reducing wrong labels in distant supervision for relation extraction. In Meeting of the Association for Computational Linguistics (ACL), 2012.
Varma, Sala, A. He, J. Ratner, and C. Ré. Learning dependency structures for weak supervi- sion models. In Proceedings of the 36th International Conference Machine Learning (ICML 2019), 2019.
Martin J Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot im- age semantic segmentation with prototype alignment. In Proceedings of the IEFE International Conference on Computer Vision, pp. 9197-9206, 2019.
Jia Xu, Alexander G Schwing, and Raquel Urtasun. Learning to segment under various forms of weak supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3781-3790, 2015.
Amy Zhao, Guha Balakrishnan, Fredo Durand, John V Guttag, and Adrian V Dalca. Data augmen- tation using learned transformations for one-shot medical image segmentation. In Proceedings of the IEEE conference compuler vision and pattern recognition, pp. 8543-8553, 2019.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 EXTENDED RELATED WORK
Coarse labels Coarser forms of annotation per image can be used to train segmentation models with less manual labor. For example, bounding boxes, scribbles, and class labels can be used to provide supervision for segmentation models (Sakinis et al.) 2019 Xu et al. 2015 Khoreva et al.) 2017). This approach is sometimes referred to as "weak supervision in the segmentation literature. We note that this technique is different than the WS approach presented here, as it still requires each training sample to be annotated and there is not unified model to aggregate multiple noisy labeling sources. We refer weak supervision as the process by which multiple noisy segmentation masks are generated and aggregated automatically.
Conditional random fields Traditional approaches to modeling segmentation masks for images use conditional random fields (CRFs). These distributions often utilize unary potentials over each pixel and pairwise potentials between pixels. Many methods
proposed for defining the unary engineering features over the color, location, and texture of regions in the image (Shotton al., 2009) well pairwise interactions between pixels, such how smooth
segmentations were expected to be (Kohli et al.] 2009] Krähenbühl & Koltun 2011). Additional work was devoted performing inference over the large CRFs, developing new methods for ap- proximate inference via belief propagation and graph cuts (Kohli et al. 2007 Boykov et al., 2001
Boykov Kolmogorov 2004).
We note that the graph structure we use (Figure 22) bears resemblance to those used in past work in CRFs. However, most traditional CRF frameworks, the parameters of the PGM are learned or tuned from labeled training data. Here, we work with a latent ground truth variable Y and generate probabilistic labels for each pixel by learning how to aggregate I.F votes without access to ground truth data.
A.2 ALTERNATIVE LABELING FUNCTIONS
We experimented with many different types of LFs in the course of this work. We focus on the one- shot models used in this work because they were general (i.e., be applied any task without requiring user-modification) and produced strong results. However, for certain tasks, additional LFs may be included in Step to provide additional signal to the PGM. For example,
Physical models. For segmentation problems where structures in all images arranged similarly, physical models (e.g., atlases) can be registered to the unlabeled images and provide weak labels via label propagation.
Pre-trained networks. If the new segmentation task is similar to commonly studiee tasks (e.g. in natural image segmentation), off-the-shelf networks may be deployed. Image features. For simpler tasks, latent image features (e.g. color, intensity) can provide weak segmentation signal. These features can also be utilized via traditional CV segmen- tation approaches such active contour, random walker, and watershed segementation. Clustering. Unsupervised segmentation methods such as SLIC, which clusters images in XYZ-color space, can be easily implemented for tasks that have distinct boundaries.
A.3 LEARNING MODEL PARAMETERS
We rely on a technical fact: our proposed distribution is minimal, so that finding the canonical parameters can be accomplished by finding the mean parameters of the model: (Ca())) E[L(",)Y""']] and C and performing the mcan-to-canonical parameter mapping. See Wainwright & Jordan 2008 for more details. We take E[1 and E {[1 as
inputs, representing the conditional class balance of all pixels and the conditional rate of agreement of neighboring pixels, respectively. We note that these values can be estimated from validation set, tuned, or estimated from the observed values of the labeling functions. The remaining mean parameters to be estimated are then E[1 [Ca) y For conciseness, we will refer to Y." as Y and xp_h A in the remainder of this section.
14

---

Published as a conference paper at ICLR 2021
The mean parameter [[1[Ca] can be interpreted as the scaled accuracy of labeling function S* under condition Cd. dz. That is, rather than being between and 1, is sealed to be between -1 and 1:
a = [[[[Ca] =
P((**
(3)
This mean parameter cannot be directly estimated (i.e., from samples of À and Y) do not observe the ground truth label Y. Instead, building off of the triplet method for parameter recovery presented in Fu et al. (2020), the conditional aceuracy of each labeling function can be efficiently computed in closed-fore using a system of equations.
We use the following decomposition factor observable quantity into the product of two condi- tional accuracies n² and a²- We take to be in the contention set C and be in the non-contention part I. The decomposition holds due to the fact that these terms are uncorrelated, which follows from the conditional independence of the underlying À variables.
Y? d' E[1 (Y
F(1 (C.)),
Having access conditionally independent labeling functions is precisely why selected two LFs and kept them in the set I, which does not overlap with the contention set D.
With the equation above, we can observe all of the terms such as 1 [ X so that we can empirically estimate the final term. If we form triplets of such equations, for the pairs (j, k), (i, E), (ks, E). where j E D and k. E I, then we can use the following system of equations to solve for the dy az, agz up to sign. Specifically, the system is
aaa E[A{Ca)A xpx)) da = C[ x.d) La E| J.
The magnitude of the solution is for j
B X[
la'al
B[ 11
In practice, we use the empirical versions of each of the expectation terms. In general, long as the accuracies are better than random, the sign is positive, that we fully recover dy. Even in cases where the signs of some accuracies are negative, it is still possible to recover in a range of conditions, using the technique detailed Ratner et al. (2019).
This procedure can be done for any of the LFs and choices of the contention. All that is required to select valid triplets. By design, there is only one possible valid grouping of triplets: each labeling function in D upon which the conditions rely is grouped with the two labeling functions in I upon which no conditions rely. We take the median conditional accuracy estimate for each labeling function in J. The division of labeling functions into I and D is chosen to maximize the number of LF disagreements caught by the conditionals in the dependent set.
With this we have recovered all of the mean parameters. To obtain the canonical parameters the 0's in 2 we use the mean-to-canonical mapping, as detailed in Wainwright & Jordan (0088)
A.4 ADDITIONAL EXPERIMENTAL RESULTS
Image preprocessing The CMR images were corrected with an N4 bias field correction and nor- malized with histogram equalization. The ACT images were preprocessed with a CT window (win- dow level 50, window width 400) and each axial slice was resized to 224x224, then normalized to the range [0.1]. The BRATS dataset was resized such that each axial slice was 224x224, then normalized to the range [0,1].
15

---

Published as a conference paper at ICLR 2021
Ablations We explore the effect of each step of our pipeline in Table below. We note that these are the Dice scores reported over the validation sets because the test set never processed by the LFs.
Effect of LF aggregation. We report the score of each of our one-shot LFs and compare it the Dice score of the aggregated LFs.
Effect of aggregation rule. We compare different aggregation rules our datasets: the majority vote of the labeling functions, our proposed label model, and Flying Squid (Fu et al. (222)). We note that the current implementation of Flying Squid unable to perform inference over the large number of pixels in the ACT and BRATS datasets.
Effect of training end model. We show the impact of training an end model with the weak labels compared to the weak labels alone.
Effect of fine-tuning. We show the impact of fine-tuning the end model with the small number of hand-labeled examples.
Table 1: Ablations.
Cardiac MR
Ab. CT
BRATS Tumor
Method
LV Endo
LV Epi
RV
Liver
Spleen
Esophagus
Labeling functions
LF
89.0 (0.7)
75.3 (0.3) 71.4 (1.7) 91.8 (0.2) 86.2 (0.6)
(3.0)
71.7 (1.8)
LF2
81.5 (1.5)
(1.9 55.1 (2.6) 92.5 (0.1) (3.3)
68.8 (3.1)
LF3
(0.5)
59.7 (0.8) 62.9 (1.1) 92.8 (0.2) 88.3 (1.2)
(3.3)
73.1 (4.0)
LF4
90.5 (0.6) 74.1 (0.6)
80.8 (1.8) 83.7 (0.9) 72.1
30.0 (1.2)
73.8 (2.8)
LF5
86.2 (1.6) 58.8 (0.5) 61.4 (1.9) 93.0 (0.6) 86.7 (0.3)
46.5 (1.3)
70.8 (5.0)
Aggregation rules
Majority vote
(0.4) (0.8) 79.1 (1.6) (0.3) 91.2 (0.1)
42.1 (0.4) (1.0
Flying squid
92.3 (0.5) 76.7 (0.6) 79.0 (2.0)
Proposed PGM
92.8 (0.2) 77.7 (0.4) 82.3 (1.2) 94.5 (0.2) 90.7 (0.5) 45.1 (2.8) 77.3 (0.6)
End model
End model
94.0 (0.2) 79.3 (0.5) 85.0 (0.9) 95.0 (0.2) 94.1 (0.2) 51.7 (0.0) 78.0 (0.8)
Fine tuning
End model + FT 94.4 (0.1) 82.4 (0.3) 87.5 (1.5) 95.7 (0.1) 94.2 (0.3) 59.5 (0.4) 77.7 (0.3)
Additional qualitative results In Figure A.1 we show qualitative visualizations the segmenta- tions described in Section 4.2.1
Generalization performance For methods that use fewer training labels, we hypothesized that the final CNNs may be less robust natural image variations than fully supervised networks. We assessed the models' abilities to generalize to new test sets for two tasks. While the new datasets are of the same structure, view, and modality the originals, they represent different patient populations and scanners. As an additional CMR test set, we collected 100 CMR images with endocardium
segmentations through collaboration with the NIH. As an additional ACT set, we use images and liver segmentations from the CHAOS challenge (Kavur et al.) 2020a 2019).
Table 2: Generalization performance.
Endo.
Liver
PANet
+3.76% (4.93%)
+6.35% (3.52%)
GAN DA
-15.00% (3.01% )
+2.07% (1.32%)
CL Proposed
-11.2%% (3.98%)
-0.69% (0.97%)
-5.8% (0.87%)
+2.48% (0.66%)
FS-all
-3.94% (0.37%)
+0.38% (0.27%)
We compare the performance of the trained models on these new test sets and report the change in performance compared to the original test in Table 2 of the Appendix. We compared the best
16

---

Published as a conference paper at ICLR 2021
Proposed
Proposed
Ground bruth
Ground truth
Proposed Ground truth
Proposed Ground
Proposed
Proposed
Ground truth
Figure A.1: Example results visualized for each dataset, starting from the top left and proceeding by row: endocardium, myocardium, right ventricle, liver, spleen, esophagus, and tumor.
performing method from the knowledge transfer approaches (PANet), the best performing method from the self- and semi- supervised approaches (CL., along with our proposed approach, the data augmentation approach, and the fully supervised network. We see that the WS approach behaves most similarly to the fully supervised networks. However, PANet actually is able to best generalize to the new test sets (relative its original performance), possibly because of its inference procedure, which utilizes labeled images from the new test set.
17


Reviews:
[{"review_id": "yq5lgFz1gJY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents weakly supervised framework for image segmentation tasks with limited annotated data. It first builds several labeling functions with limited annotation and then uses probability graph to fuse the labels.  Last, the final output will be generated via CNN network.  There are two key challenges in such setting. First, how to build the labeling functions. Second, how to measure the accuracies from the labeling functions. \n\nHowever, these two challenges are not well described in this paper. First, the labeling functions are built on prior knowledges such as physical models, clustering and pretrained network.  Especially , for the pretrained network, it is not fair in few shot learning setting.  Second, the author claims she/he uses one image to train the LFs, since applying one image to train the network is difficult, it is very confusing on how to train the network. Third,  the author uses 60% data for training, it is not clear how to differentiate the proposed weakly supervised training with traditional supervised training (it is not uses true label?but the paper not describe it).  Morever, since FS-5 has reasonable good performance, the author should also compares with FS-60%.  \n\nSince current paper has lots parts need to be cleared, the current draft can not be accepted. \n\n\n-----------------------------------------------------------------------------------------------------------------------------------\nThe author answers to reviewers' questions clearly with supportive details.  In addition,  the newer version has added useful new baselines and results. I have increased my score as all my concerns get cleared.\n\nOverall,  I think  this is an interesting paper and should be encouraged. But I still has some concerns as the application is very limited on medical images. The author has compared all the baselines and shown that the generalization capability for the proposed method is similar to fully supervised method, which seems not doing well.  I would be more convinced if the proposed can be tested on few shot benchmark which are natural images. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mMezdPk1V25", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors employ ConvNets trained with a very small set of reference labeled images, and introduce a new weakly supervised model that focuses on certain areas of the image to fuse these weak sources (i.e. their approach introduces a conditional model that focuses on areas where the labeling functions disagree, more finely tracking the performance in such areas). I think the paper is interesting and well-presented, but there are several important aspects that, in my humble opinion, harm my overall rating of the paper:\n\n- Not clear to me what is the main original contribution and novelty of this work. They use generic and simple labeling functions to obtain weak supervision and, then, they use a probabilistic model to refine the accuracy for each labeling function. As far as I knoe, both approaches (knowledge injection via weak segmentation algorithms, and probabilistic graphical models to model segmentation masks) are not novel, so what is the main methodological contribution and novelty of this work?\n\n- Why the authors do not compare with other state-of-the-art few-shot medical image segmentation approaches (like [1])? In fact, [1] is already cited in the paper, but the experimental comparison does not include it. Why is that? Without a more extensive experimental comparison with prior methods is difficult to elucidate the actual empirical contribution of the proposed method. \n[1] Ouyang, Cheng, et al. \"Self-supervision with Superpixels: Training Few-Shot Medical Image Segmentation Without Annotation.\" ECCV 2020.\n\nThere are other details that should be clarified or improved in the paper:\n\n- Are the authors employing one (\"we use one-shot learning: we train a CNN from a single labeled data point for each LF.\") or five (\"five images are annotated by hand and used to seed LFs that weakly label the rest of the training set.\") labeled images to weakly label their training data?\n\n- The conclusions of the paper are too succint (just two lines and a half). I would encourage the authors to extend this section. \n\n- What fully supervised network (\"FS-all\") do the authors use in their experiments?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Git25vg8j_c", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "CUT OUT THE ANNOTATOR, KEEP THE CUTOUT: BETTER SEGMENTATION WITH WEAK SUPERVISION\nThe authors make a clear explanation of the problem and rationale behind their approach, but a few experimental questions should be addressed concerning the implementation of the models and needs some additional results.\n\n*Review\nThe paper is well written with significant time spent setting up a clear explanation of the background information and challenges.\nThe authors also adequately discuss alternate approaches and other methods currently used to address this current problem.\nThe raw numeric results of the proposed method are very compelling.\n\nThe authors make a good case for the significance of this work if it is able to surpass current methods in this space.\nThere are several questions that are still left un-addressed:\n\n-Because the size of a neural network impacts the quality of the segmentation, it is important to know the size of the U-Net that they used for each model. The proposed method uses several CNNs in combination. So depending on the size of networks used in the proposed method (compared to the network size of the other chosen methods), the results might be biased based solely on the size of the networks used.\n\n-The authors explain that the stacked networks will not add computational time due to the fact that they are each trained on only 1 image. This might be small since they were only given 5 training images, but this time will scale with more training data. Also, when stacked networks are applied to large datasets at test time, the additional computational times will add up. It would be nice to see a numeric comparison of computational times.\n\n-One problem of learning from small datasets is guaranteeing balance and the quality of the training images provided (Changing out even a single image can impact model training). How variable are the results on what training images are provided? Authors state that they trained the models three times with different seeds, but it is assumed they used the same training images in every iteration.\n\n-Authors state that random image augmentations were used in all models except FS-5 (unstated what was used for FS-all). Since image augmentation is important to apply to test set and generalization in general, could authors provide the rationale as to why baseline models weren’t trained with image augmentation?\n\n- In Table 1 three trained models were averaged. It would be nice to also show deviations alongside the means so we can see the consistency of each approach.\n\n- In Figure 4. Scaling of the axis could be misleading. Keep colors consistent for the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DtarLGjro5O", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper studies few-shot segmentation for medical images where labeled data is hard to obtain.  It proposed a stage-wise pipeline where pseudo labels are first proposed by noisy learning functions (LFs), then aggregated through a PGM, and finally used to train a segmentation model. The authors argue the two biggest challenges are \"injecting knowledge for segmentation model\" and \"measuring accuracies of pseudo labels\", and propose two modules to address them. The experiments have shown that the proposed method outperforming several few-shot learning baselines.\n\n### Pros\nThe studied application is indeed very important and requires few-shot learning techniques. The 2nd challenge (\"measuring accuracies of pseudo labels\") is very insightful and indeed a huge problem for many methods. The technical solution of using PGMs is interesting and should be encouraged.  The experimental results are promising and encouraging.\n\n### Cons\n1. The abbreviations are abused, especially about FSL, LLL, and WS. As a reviewer from vision community, there is a common agreement of the definition of FSL and WS. LLL is rarely used and usually represents a very broad concept rather then anything specific. This paper has different interpretations of these concepts, which is fine, but should make it really clear what they stands for and what are the differences between them. In my opinion, this paper is simply a FSL setting and there are no need to introduce other two terms. \n2. The notations are also not super clear to me. In Sec.3.1, it's said $X_i \\in \\mathbb{R}^{I,J,K,T}$ is a tensor of $P$ pixels. However, $P$ doesn't go into any places of the notation. Moreover, what are $I,J,K,T$? They are never properly introduced. Also,  the superscript and subscript in $X_{i=1}^{i=n}$, $Y_{i=1}^{i=n}$ are just self-contradictory. \n3. In Sec.3.2 four LFs are discussed (Physical models, pre-training networks, ...). Are these methods ever used in the proposed framework, or compared to as baselines in experiments? Also, the reference of these methods are missing.\n4. It seems like I miss something. There seem to be only one label function:  a few-shot image classifier. What are the multiple LFs? \n5. The design choices are not properly ablated. It's unclear which part contributes the most.\n6. The FSL baselines are very limited: only PANet. In fact there are many orthogonal solutions of FSL. It's preferable multiple different baselines are compared  for this relatively new task.\n7. Tab.2 results are obtained by directly comparing accuracy numbers for two different dataset, which doesn't make sense to me as they are not comparable. A better way to evaluate generalization performance is to compare \"test a model pretrained using A on B\" and \"test a model trained on B\". \n\nI have some doubts about the evaluation:\n1. What's the reason for using Dice as evaluation metric rather than standard IoU, Pixel-Acc metrics? DICE is not popular for any segmentation work, and from the formulation it seems to be a worse metric than IoU (union area summed twice in the denominator).\n\nI have some doubts about the motivation:\n1. Why \"Injecting knowledge via LFs\" is a big challenge? Isn't it the standard pseudo-label or knowledge distillation algorithm? Multiple semi-supervised/weakly-supervised segmentation papers have studied this idea. To name a few:\n*. Papandreou and Chen etal., Weakly- and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation. ICCV 2015\n*. Wei etal., STC: A simple to complex framework for weakly-supervised semantic segmentation. PAMI\n\n### Misc\n1. Since all the experiments have been runned 3 times and the labeled samples are few (5), it's worth reporting the variance/std together with the mean.  Some results in Tab.1 are very close and hard to tell which method is better. \n2. Sec.3.2 `` \"LLL labeling functions\" writs \"one-shot learning\". However, in experiments it seems like 5 labeled images are used. Better to be rigorous.\n3. What's the dimension of $\\theta_i$ in Eq.(1)?\n4. Fig.4 can be improved to be more consistent via using the same color for proposed method.\n5. I would change Fig.1 example to use medical images.\n\n### Questions\n1. Have the authors tried to sample different labeled images? Will the results be different? Compared to using random seeds, I would report mean results over different samples.\n\n********************************\nThe rebuttal is very detailed and contains many useful new baselines and results. The authors answer to reviewers' questions carefully and with great supportive details. I appreciate the efforts and many of my questions are answered, therefore I have increased my score. \n\nI'm not  an expert from medical image field. So it's a little bit hard for me to evaluate the significance of the reported results and proposed methods. However, I think this paper is well-motivated and current version has greatly improved over the first draft.  For the machine learning community, it's an interesting and important application and thus should be encouraged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9s_MmRRUwr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of performing medical image segmentation in the limited label scenario, i.e. the case where there is only a small number of manual expert segmentations available.\n\nThe proposed approach to address this challenge uses a combination of two approaches: First, multiple labelling functions are learned by training CNNs with a single labelled example. The resulting labelling functions are then combined using a label fusion approach. This label fusion approach is based on a conditional model that focuses on areas of contention. To the best of my knowledge, the proposed approach is novel.  I found the methodology interesting and well motivated. The formulation and writing is clear. The main weakness to me is the evaluation. First, the selected baselines are not always very strong. For example, the best performing method for segmentation of ACDC (cardiac MRI) reported in Bernard et al., 2018 achieves significantly higher results than the baseline segmentation reported (”FS-all”). \n\nThe authors mention that, depending on task difficulty, more sophisticated limited label learning models could be used as labelling functions. It would be good to see some discussion of this.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article proposes a weakly supervised few-shot learning method for medical imaging segmentation. While initially, the article presented several problems indicated by the reviewers, e.g., the explanation of the novelty and contributions, the explanation of the method, and the experimental evaluation, the authors made a great effort addressing most of the reviewers' comments and uploaded an updated version of the article. However, still, the evaluation part of the article is a bit weak. But the article contains interesting contributions. Accordingly, I recommend accepting the paper at ICLR2021.
Decision: Accept (Poster)