A multi-task U-net for segmentation with lazy labels
Authors: Rihuan Ke
Aurélie Bugeau
Nicolas Papadakis
Peter Schuetz
Carola-Bibiane Schönlieb
Venue:  (2020)
Keywords: multi-task learning
weak labels
semisupervised learning
image segmentation

Abstract: The need for labour intensive pixel-wise annotation is a major limitation of many fully supervised learning methods for image segmentation.   In this paper,  we propose a deep convolutional neural network for multi-class segmentation that circumvents this problem by being trainable on coarse data labels combined with only a very small number of images with pixel-wise annotations.  We call this new labelling strategy ‘lazy’ labels.  Image segmentation is then stratified into three connected tasks: rough detection of class instances, separation of wrongly connected objects without a clear boundary, and pixel-wise segmentation to find the accurate boundaries of each object. These problems are integrated into a multi-task learning framework and the model is trained end-to-end in a semi-supervised fashion. The method is demonstrated on two segmentation datasets, including food microscopy images and histology images of tissues respectively. We show that the model gives accurate segmentation results even if exact boundary labels are missing for a majority of the annotated data.  This allows more flexibility and efficiency for training deep neural networks that are data hungry in a practical setting where manual annotation is expensive, by collecting more lazy (rough) annotations than precisely segmented images. 

Under review as a conference paper at ICLR 2020
A MULTI-TASK U-NET FOR SEGMENTATION WITH LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
The need for labour intensive pixel-wise annotation is a major limitation of many fully supervised learning methods for image segmentation. In this paper, we propose a deep convolutional neural network for multi-class segmentation that circumvents this problem by being trainable on coarse data labels combined with only very small number of images with pixel-wise annotations. We call this new labelling strategy 'lazy' labels. Image segmentation is then stratified into three connected tasks: rough detection of class instances, separation of wrongly connected objects without a clear boundary, and pixel-wise segmentation to find the accurate boundaries of each object. These problems are integrated into multi- task learning framework and the model is trained end-to-end in semi-supervised fashion. The method is demonstrated two segmentation datasets, including food microscopy images and histology images of tissues respectively. We show that the model gives accurate segmentation results even if exact boundary labels are missing for majority of the annotated data. This allows more flexibility and efficiency for training deep neural networks that are data hungry in a practical setting where manual annotation is expensive, by collecting more lazy (rough) annotations than precisely segmented images.
INTRODUCTION
Image segmentation has been an active research field in the past decades. Deep learning approaches play increasingly important role and have become state-of-the-art in various segmentation tasks (Huang et al.) 2018 Khoreva et al. [017 Tsutsui al. 2018 8. Ghosh et al., 2018 Litjens et al. 2017). Though fully supervised segmentation neural networks have a shown great success, one of their most challenging issues is the need for pixel-level annotations to train them. Obtaining such annotations usually requires great amount of manual work and is therefore expensive.
In this paper, we propose multi-class and multi-instance segmentation approach that we split into three relevant tasks: detection, separation and segmentation (cf. Figure []. Doing so, obtain cun--supervised learning approach that is trained with so-called "lazy" labels, that is a lot of coarse annotations of class instances together with only few pixel-wise annotated images that be obtained from the coarse labels in a semi-automaae way. In the following, we will refer to weak (resp. strong) annotations for coarse (resp. accurate) labels and denote them as WL and SL.
Task 1 detects and classifies each object and roughly determines its region through an under- segmentation mask. Instance counting can be obtained objective is instance detection, exact labels for the whole object or its boundary are not necessary at this stage. We use instead weakly annotated images in which rough region inside each object is marked, cf. the most top left part of Figure [ For segmentation problems with dense population of instances, such as the food components (see e.g., Figure TD. cells (Guerrero- Pena et al. 2018 Ronneberger et al. 2015), glandular tissue, people in crowd (Wang et al. (018b), separating objects sharing a common boundary a well known challenge. We can optionally perform a second task (Task 2) that focuses on the separation of instances that are connected without a clear boundary dividing them. Also for this task we rely on WL to reduce the burden of manual annotations: touching interfaces specified with rough scribbles, cf. top left part of Figure I Task 3 finally tackles the pixel-wise classification of the instances. It requires strong annotations that are accurate up to the boundaries of the objects. Thanks to the information brought by the weak annotations, here just
by-product of this task. As the main

---

Under review as a conference paper at ICLR 2020
Lazy labels
Weak annotations
Input
crystals
Separations
Training
Multi-Task U-net
Strong annotations
Segmentation
Piselwise classilication
crystall
Figure 1: Multi-task learning for image segmentation with lazy labels. The figure uses Scanning Electron Microscopy (SEM) images of food microstructures as an example and demonstrates a segmentation problem of three classes, namely air bubbles (green), ice crystals (red) and background respectively. Most of the training data are weak annotations containing (i) partial marks of ice crystals and/or air bubbles instances and (ii) fine separation marks of boundaries shared by different instances. Only few strongly annotated images are used. On the top right several SEM images are displayed. Their corresponding output, obtained with the trained network, are shown the bottom right.
need a very small set of accurate segmentation masks, cf. bottom left part of Figure To that end, we propose to refine some of the coarse labels resulting from task 1 using a cemi-automatic segmentation method which requires additional manual intervention.
The three tasks are handled by a single deep neural network and are jointly optimized. The network architecture is inspired by the widely used segmentation network named U-net (Ronneberger et al. 2013). With all three tasks sharing the same contracting path, we introduce new multi-task block for the expansive path. The network has three outputs and is fed with combination of WL and SL described above. Since weakly and strongly annotated training data is shared between the tasks, part of the annotations are missing, especially for task 3. To accommodate for this we introduce weighted loss function over the samples. Accurate segmentation labels for training are usually not easy to obtain, however, with our approach we demonstrate that exact labels for the whole training set are not needed for good segmentation learning performance.
We evaluate the performance of the proposed approach on two applications, namely for the segmenta- tion of SEM images of food microstructure and stained histology images of glandular tissues.
In summary, our contributions are as follows. (1). We propose a decomposition the segmentation problems into three tasks and corresponding user friendly labeling strategy. (2). We develop multi-task learning framework that learns directly from the manual labels and trained end-to-end Our approach outperforms state-of-the-arr weakly supervised methods such as Khoreva et al. (2017) (3). The network predicts segmentation mask as well as the object inner regions and touching object interfaces. Touching objects are effectively disconnected side product.
RELATED WORK
In image segmentation problems, one needs to classify an image at pixel level. It is a vast topic with diversity of algorithms being developed, including traditional unsupervised methods like k--maans that splits the image into homogeneous regions according to image low level features, curve evolution based methods like snakes (Caselles et al. 1997), graph-cut based methods like Grabeut (Rother et al. 2004), just to name few. Interactive approaches such snakes Grabcut enable getting involved users' knowledge by means of initializing regions or putting constraints on the segmentation results.
Deep convolutional neural network (DCNN) approaches have been developed for segmenting complex images, especially in the semantic setting. In particular, fully convolutional networks (FCN) (Long et al.) 2015) replace the last few fully connected layers of conventional classification network by up up-sampling layers and convolutional layers, to preserve spatial information. FCNs have many variants for semantic segmentation. The Deeplab (Chen al. 2018) uses technique called
2

---

Under review as a conference paper at ICLR 2020
atrous convolution to handle spatial information together with fully connected conditional random field (CRF) (Chen et al. 2014) for refining the segmentation results. Fully connected CRF can be used post-processing can be integrated into the network architecture, allowing for end-to-end training (Zheng et al. 2015).
Other types of FCNs are deep encoder-decodes networks (Badrinarayanan al. 2017 Ronneberger et al. 2013). They have multiple up-sampling layers for better localizing boundary details. One of the most well-known models is the U-net (Ronneberger et al. 2015). It is fully convolutional network made of a contracting path, which brings the input images into very low resolution features with a sequence of www.saapling.....m.m and an expansive path that has an equal amount of up up-sampling layers. The higher resolution feature maps on the contracting path are merged with up-sampled layers via long skip connections to recover boundary information after wown-sampling.
wetpld///emm-uuuervissee segmentation learning, Motivated by the heavy cost of pixel-level anno- tation needed for fully supervised learning, there has been growing interest in weakly supervised learning with coarse annotations for semantic segmentation in the last years. Common weak anno- tations include image-level labels (Huang et al. 2018 Papandreou et al. 2015; Lee et al. 2019),
bounding boxes(Khoreee et al. 2017), scribbles (Lin et al. (016) and points (Bearman et al. 2016
Most weakly supervised deep learning methods for segmentation are built on top of a classification network. The training of such networks may be realized using segmentation masks explicitly gen- erated from weak annotations (Wei et al., 2017: Khoreva et al.] 2017 Lee et al. 2019: Wei et al.) 2018 Tsutsui al. 2018). The segmentation masks be improved recursively, which involves several rounds of training of the segmentation network (Wei et al. 2017 Jing et al. 2018 Ezhov
al. 2018
Other techniques do not use any explicit pseudo segmentation masks as training data, but instead compute a composite loss from other guiding principles. For instance, the SEC method (Kolesnikov & Lampert 2016) combines localization cues (Seed), classification coherence (Expand) and Constrain- to-boundary with fully-connected CRE. Special modules could be designed to further exploit the information of sources, such as the Deep Seeded Region Growing (Huang et al. 2018) and saliency seeded region growing (Sun Li 2019).
Semi-supervised deep learning provides an alternative way to reduce the pixel-level annotation burden (Baur et al., 2017 Perone& & Cohen-Adad 2018). These methods require only a small amount of strongly annotated data and a large set of unlabelled data. Several training methods that combine weak annotations with a limited set of strong annotations have already been explored (Papandreou eet al.. 2015 Wei et al. 2018: Lee et al.) 2019). The pixel-wise labels can be integrated through an additional loss function, and used along with their semi-supervised counterparts. Better weak supervision performance can be obtained by feeding the network with learned object localization maps (Wei et al.] 2018 Lee et al. 2019). However, the networks are trained with image level labels from large scale datasets like ImageNet (Deng et al., 2009) or PASCAL VOC 2012 (Everingham et al. 2015).
The weakly and semi-supervised deep learning methods are also explored for object localization or segmentation on relatively smaller datasets, especially in medical imaging. For these methods, the weak annotations can be of image level (Mlynarski al. 20188 Playout et al. 2019 Zhou al.
2019 Shin et al. 2019) or in forms of bounding boxes (Shah et al. (018).
Multi-task learning. Multi-task learning algorithms consider several related tasks improve the overall performance, taking benefits from the underlying common information that may be ignored by a single task learning. deep neural based multi-task models, the common information conveyed by soft or hard parameter sharing (Ruder) 2017). Various multi-task deep learning methods have been developed for segmentation, for example, the stacked U-net for extracting roads from satellite imagery (Sun et al. 22018), the two stage 3D U-net framework for 3D CT or MR data segmentation (Wang ctt al. Sa), encoder decoder networks for depth regression, semantic and instance segmentation (Kendall et al. 2018), and for building footprint segmentation (Bischke al. 2017). Compared to these works, our method handles the tasks with a multi-task block at different feature resolutions and designed upon both strong and weak notations.
Our work is more closely related to the multi-task learning methods in PPlyyout et al. 2018 2019)
for retinal lesions segmentation and (Mlynarski et 2018) for brain tumour segmentation. Three tasks are considered in (Playout et al. 2019): red lesion segmentation, bright lesions segmentation
3

---

Under review as a conference paper at ICLR 2020
and image level lesion detection. The proposed encoder-decoder architecture has one branch in the encoder part and two branches into decoder part specified for the red/bright lesion segmentation and the image level classification respectively. The network is weakly supervised as only image level labels are used in one of the training phases. In (Mlynarski et al. 2018), a U-net architecture is used for jointly segmenting and classifying brain tumours. An additional branch takes the last but one layer of a standard segmentation U-net as input, followed by a mean pooling layer, and outputs a score for the classification task.
Our network structure, similar to these architectures, shares an encoder part for all tasks. The learning is also supervised by mixture of strong annotations and weak annotations. However, the proposed method does not include supervision from image level annotations and is more specialized in distinguishing the different object instances and clarifying their boundaries in every single image.
MULTI-TASK LEARNING FRAMEWORK
Fully supervised learning for segmentation approximates the conditional probability distribution of the segmentation mask given the image. Let s be the segmentation mask and / be the image, then the segmentation task aims estimate p (3) based set of sample images I {1, Iz, In..
(.")). The set I is randomly drawn from an unknown distribution. In our setting, having the whole set of segmentation labels {s h.... is impractical, and we introduce two auxiliary tasks for which the labels can be more easily generated to achieve an overall small cost on labeling.
and the corresponding labels {a(") 8/2
For given image I E I, we denote as (2) the rough instance detection mask, (22) a map containing some interfaces shared by touching objects. To create s(*), the contours of the objects are not treated carefully, resulting in coarse mask that misses most of the boundary pixels, cf, the left of Figure [ Let I, C I denote the subset of images labelled for task &e (k = 1, 2,3). As we collect different amount of annotations for each task, the number of annotated images [Tx] may not be the same for different Typically the number of images with strong annotations [Tal << TI.
The set of samples in Ia for segmentation being small, the computation of an accurate approximation of the true probability distribution p ( ()) is a challenging issue. Given that much more samples of
(1)) and (2)) are observed, it relatively easier to learn the statistics of the weak labels. Therefore, in multi-task learning setting, one aims at approximating also the conditional probability p(5 p(x(11
and p( (s(2) 1) for the other two tasks, or the joint probability (((("), (22), s.()) |1). The three tasks are connected to each other. By the definition of the detection task, it is not difficult to see that p(s(()) = 2 s1) = x) = 0 for æ and z satisfying 2i,c = 1 and Z,,e = O for some pixel and class c other than the background. The map of interfaces (2) indicates small gaps between two connected instances, and is therefore a subset of boundary pixels of the mask s(3)
Let us consider the probabilities given by the models p(a(t)) (;0) 1,, parametrized by 0 which is determined such that the models match the desired probability distributions. The parameter 0 is shared among all the tasks. We do not optimize 0 for each individual task, but instead consider (1:0) Assuming that (1) (rough under-segmented instance detection) and (22) (subset of shared boundaries) are conditionally independent given image 7, and if the samples are i.i.d, we define the maximum likelihood (ML. estimator for 0 as
a joint probability ((s!"), (s("), g2)) .(3))
OML arg IILAX 0
(3) log 8
.1:0) + log pl (K) (1:0)
(I)
TET
The set Ia may not be evenly distributed across I, but we assume that it is generated by fixed can be approximated correctly
distribution as well. Provided that the term {0( (199) ()) s12 })} fEz
by p( (5(3) (1) (2) .1:0 even if 0 is computed without (3) specified for I'Z,, then
logpl 8
s (1) (2) 1:00 x log (3)
1 (1) 8 1:0)
(2)
I024
4

---

Under review as a conference paper at ICLR 2020
If furthermore, the segmentation mask does not depend on (2)) or (2) given I E Ia, and if [Za] [Ta] are large enough, then from Equations ① and (2), we approximate the ML estimator by
arg max
Σ IEIx
log logo(st (1 (1:0)
(3)
in which 01, 022 0g negative constants.
3.1 LOss FUNCTION
Let the output of the approximation models be denoted respectively by h. (4). (1), (g)()), and hg (I),
with hng (T) L.
the estimated probability of pixel being in class of task k. For the label (b)) of the log likelihood function for each task is decomposed into
log 80 1:0) Σ
Σ
$,. log
( (T) k.e" = 1,2,3,
(4)
CECP
in which (K) denotes the element of the label (A)) at pixel for class e and C2 is the set of classes for task k. For example, for ice cream images, have three classes including air bubbles, ice crystals and the rest (background or parts of the objects ignored by the weak labels), so C2, C3 = {1, 3). For the separation task, there only two classes for pixels (belonging or not to touching interface):
C2 = {1, 2}. According to Equation 3), the network is trained by minimizing the weighted cross entropy loss:
(0) =
a o I, lo) ( TT:)):
(5)
k=
Here 1z, ()) is an indicator function which is if E Is and 0 otherwise.
3.2 MULTI-TASK NETWORK
We follow a convolutional encoder-decoder network structure for the multi-task learning. The network architecture is illustrated in Figure As extension of the U-net structure for multiple tasks, we only have one contracting path that encodes shared features representation for all the tasks. On the expansive branch, we introduce multi-task block at each resolution to support different learning purposes (blue blocks in Figure 2) Every multi-task block runs three paths, with three inputs and three corresponding outputs, and consists of several sub-blocks.
In each multi-task block, the detection task (task 1) and the segmentation task (task 3) have a common path similar the decoder part of the standard U-net. They share the same weights and use the same concatenation with feature maps from contracting path via the skip connections. However, we insert an additional residual sub-block for the segmentation task. The residual sub-block provides extra network parameters to learn information not known from the detection task, e.g. object boundary localization. The path for the separation task (task is built on the top of detection/segre segrentttiintnnntiiiiinnnn ones. It is also a U-net decoder block structure, but the long skip connections start from the sub-blocks of the detectiionksemmnnatiin paths instead of the contracting path. The connections extract higher resolution features from the segmentation task and use them in the separation task.
To formulate the multi-task blocks, let au and z denote respectively the output of the detection path and segmentation path at the multi-task block 1, and let ei be the feature maps received from the contracting path with the skip connections. Then for task and task we have
0:+1 = Fw; (t),
2
= Fw; (zi, et),
Z1+1 2 + FW,+ (2444):
(6)
in which Wi, W/+1/2 0 are parameters of the network and Fw,, Fw, determined respectively by a sequence of layers of the network (Cf. the small grey blocks on the right of Figure 2 For task 2 the output at 7h block your computed by 4 = Gw, (zc41, yc) with additional network parameters Wi E 0. Finally, after the last multi-task block, softmax layers are added, outputting probability map for each task.
Implementation details. We implement a multi-task U-net with levels of spatial resolution and input images of size 256 256. A sequence of down-sampling via max-pooling with pooling
5

---

Under review as a conference paper at ICLR 2020
Input
Taoc Detection
Tank Suporation
Task Begmertation
Multi-Task Block
Oig Mest Real
Figure 2: Architecture of the multi-task U-net. The left part of the network a contracting path similar to the standard U-net. For multi-task learning, we construct several expansive paths with specific multi-task blocks. At each resolution, task (Detection yellow) and task 3 (Segmentation in red) run through common sub-block, but the red path learns additional residual to better localize object boundaries. Long skip connections with the layers from contracting path are built for yellow/red paths via concatenation. Task 2 (Separation, in green) mainly follows separated expansive path, with its own up-sampled blocks. A link with the last layer of task 3 added via skip connection in order integrate accurate boundaries in the separation task.
size x used for the contracting path of the network. Different from the conventional U-net (Ronneherger et al. 2015), each small gray block (see Figure consists of convolution layer and batch normalization (loffe & Szegedy, 2015). followed by a leaky ReLU activation with a leakiness parameter 0.01. The same setting also applied to gray sub-blocks of the 4 multi-task blocks. On the expansive path of the network, feature maps are up-sampled (with factor 2 x 2 by bilinear interpolation from low resolution multi-task block to the next one.
3.3 METHODS FOR LAZY LABELS GENERATION
We explain our strategy for generating all the lazy annotations that used for training the multi-task U-net. We introduce our method with a data set of ice cream SEM images but any other similar dataset could be used. Typical images ice cream samples are shown in the top row of the left part of Figure S The segmentation problem is challenging since the images contain densely distributed small object instances (i.e., air bubble and ice crystals), and poor contrast between the foreground and the background. The sizes of the objects can vary significantly in single sample. Textures on the surfaces of objects also appear.
As first step, bribble-based labelling is applied to obtain detection regions of air bubbles and ice crystals for task This can be done in a very fast way as no effort put on the exact object boundaries. We adopt lazy strategy by picking out an inner region for each object in the images (see e.g., the second row the left part of Figure 3). Though one could get these rough regions as accurate as possible, we delay such refinement task for better efficiency of the global annotation process. Compared to the commonly used bounding box annotations in computer vision tasks, these labels give more confidence for particular part of the region of interest.
In the second step, we focus on tailored labels for those instances that are close one to each other (task 2), without clear boundary separating them. Again, we use scribbles mark their interface. Examples for such annotations are given in Figure 3 (top line, right part) The work can be carried out efficiently especially when the target scribbles have a sparse distribution. On the other hand, as no labelling is needed for the objects that are well separated, we can collect sufficient labelled images in a limited amount of time and cover the complex of tasks and 2 are done independently. It follows the assumption made in Section that (1) and (2) conditionally independent given image T.
cream sample conditions. Lazy manual labeling
The precise labels for task 3 are created using interactive segmentation tools. Starting from the rough (inner) regions of task reached. This can be done with geodesic active contours (Caselles et al. 1997). Unfortunately, such a method fails capture sharp corners and the contour evolution tends to ignore boundaries with low
natural idea is to let these regions grow and stop when the boundaries

---

Under review as a conference paper at ICLR 2020
contrast. The annotation then requires frequent and time consuming user interaction. Instead, Grabcut (Rother et al. 2004 Hong et al. (015) a graph-cut based method. The initial labels obtained from the first step give good guess of the whole object regions. The Grabcut works well on isolated objects. However, it gives poor results when the objects are close to each other and have boundaries with inhomogeneous colors. As corrections may be needed for each image, only few images of the whole dataset are processed. A fully segmented example is shown in the last row of Figure]
4 EXPERIMENTS
In this section, we demonstrate the performance of our approach using two different datasets. For both datasets
strong labels (SL) well as weak labels (WL). We prepare the labels and design the network as described in Section [3] The overall method summarized with the 2 procedures presented in Algorithm (see the appendix).
4.1 SEM IMAGES ICE CREAM
Scanning Electron Microscopy (SEM) constitutes the state-of-the-art for analysing food microstruc- enables the efficient acquisition of high quality images for food materials, resulting into huge amount of image data available for analysis. However, to better delineate the microstructures and provide exact statistical information, the partition of the images into different food components and instances is needed. The structures of food, especially soft solid materials, are usually complex which makes automated segmentation difficult task. Some SEM images of ice cream our dataset shown the top right of Figure [ A typical ice cream sample consists of air bubbles, ice crystals and a concentrated unfrozen solution. In most situations, the air bubbles and ice crystals appear foam in the images, while the solution fills the gaps between them. We treat the solution the background and aim at detecting and computing a pixel-wise classification for each air bubbles and ice crystals instances.
The set of ice-cream SEM dataset consists of 38 images that are split into three sets (53% for training, 16% for validation and 31 % testing respectievly). Each image contains rich set of instances with an overall number of instances around 13300 for classes (ice crystals and air bubbles). For comparison, the PASCAL VOC 2012 dataset has 27 27450 objects total for 20 classes.
For training the network, data augmentation is applied to prevent over-fitting. The size of the raw images is 960 1280. They are rescaled and rotated randomly, and then cropped into input size of 256 x 256 for feeding the network. Random flipping is also performed during training. The network trained using Adam optimizer (Kingma & Ba 2014) with learning rate 2 10 and a batch size of 16.
In the inference phase, the network outputs for each patch a probability map of size 256 x 256. The patches are then aggregated to obtain probability map for the whole image. In general, the pixels near the boundaries of each patch are harder to classify. We thus weight the spatial influence of the patches with a Gaussian kernel to emphasize the network prediction at patch center.
Images
Task 2
Task 1
Task 3
Figure 3: Example of annotated images. Some of the annotations are missing because all images are labelled for task and task 2. The marks in red are for air bubbles and the ones in green are for ice crystal instances. The blue curves on the third row are labels for interfaces of touching objects.
We now evaluate the multi-task U-net and compare it to the traditional single task U-net. The performance of each model is tested on 12 images, and average results shown in Table In the table, the dice score for a class is defined as d. = 2 2 wi,8hhee (E, ai.e + E; Hi.e) where æ is the computed segmentation mask and y the ground truth.

---

Under review as a conference paper at ICLR 2020
0.98
The models
air bubbles
ice crystals
Overall
U-net on WL
0.725
0.706
0.716
U-net SL
0.837
0.794
0.818
PL approach Multi-tass task U-net
0.938
0,909
0,924
0.953
0,931
0.944
Table 1: Dice scores of segmentation results the test
0.85
images of SEM images of ice cream dataset.
WT bubbles
Ice crystala
Figure 4: The error bars for the PL and multi-task U-net, each computed from 8 different experiments
We train a single task U-net (i.e., without the multi-task block) on the weakly labelled (task 1), with the 15 annotated images. The single task U-net on weak annotations gives overall dice score at 0.72, the lowest one among the three other methods tested. One reason for the low accuracy the single task U-net weak (inaccurate) annotations is that in the training labels, the object boundaries are mostly ignored. Hence the U-net is not trained to recover them, leaving large parts of the object not recognized. Second, we consider strong annotations as training data, without the data of the other tasks, iee. only 2 images with accurate segmentation masks are used. The score the U-net trained on SL is only 0.82, which is significantly lower than the 0.94 obtained by the multi-task U-net.
We also compare our multi-task U-net results with one the major weakly supervised approaches that makes use of pseudo labels (PI.) (see e.g.. Khoreva et al. (2077): Jing et al.) (2018)). In these approaches, the pseudo segmentation masks are created from WLs and are used to feed a segmentation network. Following the work of Khoreva al. (0117), we use the Grabcut method PLs, and in our setting the partial masks are used to initialize the Grabcut, For the small subset of images that are strongly annotated, the full segmentation masks are used instead of PLs. The PL are created without human correction, and then used for feeding the segmentation network. Here we use the baseline single task U-net as the segmentation network in order to make comparison with the multi-task U-net.
create the
Our multi-task U-net outperforms the PI. approach as shown in Table 4 the error bars for the two methods with dice score collected from 8 different runs. A significant limitation of the PL method here is that its performance relies on the tools used for pseudo segmentation mask generations. Having common errors, instead of random ones, the object boundaries in the training data, the segmentation network of PL also learns to have those patterns in the prediction. The images in the left part of Figure 5 show that the predicted label of an object tends to merge with some background pixels when there are edges of another object nearby. Similar errors from the GrabCut illustrated in the appendix.
Besides the number of pixels that are correctly classified, the separation of touching instances is also of interest. In additional to the dice scores in Table we study the learning performance of multi-task U-net on task 2, which specializes in the separation aspect. The test results on the 12 images give an overall precision of 0.70 of the detected interfaces, while 0.82 of the touching objects are recognized. We show some examples of computed separations and ground truth in the right part of Figure 5 For the detection task, the network predicts probability map for the inner regions the object instances. An output of the network is shown in Figure 6 With partial masks as coarse labels for this task, the network learns identify the object instance as shown in the figure.
We finally consider the work of Bearman et al. (2016) that investigate the cost related to different types of annotations. Based on the data reported (Bearman et al., 2016) and our estimated annotation time, the collecting of WL for detection is considerably (more than 6x) faster than obtaining SL. For a fair comparison with the baseline U-net we use a larger ratio of SL for the single task learning accordingly (since no WL is used here), and the results reported in Table The proposed method still outperforms the U-net by a large margin on similar annotation time budgets, and additional SL after the first 10% do not help significantly.
8

---

Under review as a conference paper at ICLR 2020
multi-task U-net
PL
multi-task U-net separation
true separation
Figure 5: Segmentation and separation results (best viewed color). Left: the computed contours are shown in red for air bubbles and green for ice crystals. While multi-task U-net and PL supervised network both have good performance, PL misclassifies the background near object boundaries. Right: Examples of separation by the multi-task U-net and the ground truth.
Figure 6: The image (left), the predicted probability map (middle) from the detection task, and the ground truth segmentation mask (right). The red and green on the middle and right images stand for air bubble and ice crystals respectively.)
4.2 && IMAGE DATESET FOR GLAND SEGMENTATION
We also apply the approach to the segmentation of tissue histology images. In this experiment, use the GlaS challenge dataset (Sirinukunwattana et al. 2017) that consists of 165 H&E stained images. The dataset split into three parts, with 85 images for training, and 60 for offsite test and 20 images for onsite test (we will call the latter two sets Test part A and Test part B respectively in the following).
Apart from the SL available from the dataset, create set of weak labels for the detection task and separation task (as illustrated in Appendix, Figure IT). These weak labels together with a part
methods
labels
dice
multi-task
10% SL. WL
0.944
Annotation budget
single task
20% SL
0.882
multi-task
20% SL. WL
0.948
Annotation budget
single task
30% SI.
0.913
multi-task talt
50% SL + WL
0.949
Annotation budget
single
75% SI.
0.940
Table We compare the two methods under similar annotation time budgets. In each budget, two different combinations of SL. and WI. that take similar annotation time are used. The WL in this table contains 75% labels for the detection task and 100% labels for the separation task. From budget to budget 3, we increase the amount of labels (that means more annotation time is needed) in the training data, and the dice score is reported for each case.

---

Under review as a conference paper at ICLR 2020
SL Ratio
2.4%
4.7%
9.4%
100%
Ours
0.866
0.889
0.915
0.921
Test Part A
U-net 0.700
0.749
0.840
0.915
MDUnet
0.920
Ours 0.751
0.872
0.904
0.910
Test Part
U-net
0.658
0.766
0.824
0.898
MDUnet
0.871
Table 3: Average dice score for segmentation of gland. Our method uses both SL and WL. The ratio of strong labels (SL) increased from 2.4% to 100%, and the scores the methods are reported here for two parts A and B of the test sets, as split in Sirinukunwattana et al. (2017).
(c) multi-task
(d) U-net
(a) ground truth
(b) U-net (100 % SL.
9.4% greent 4.7% S1.) (ted: 9.4% St., greent 4.7%
Figure 7: Segmentation results on the gland dataset (best view in color). The ground truth and the results. For (c) and (d), Red contour denotes the results from 9.4% strong labels; Green contour denotes results from 4.7% strong labels
of the strong labels are used for training the multi-task U-net as illustrated Algorithm T (in the appendix).
In this experiment, we the algorithm on different ratios of SL, and compare it with the baseline U-net, and the Multi-scale Densely Connected U-Net (DMUnet) (Zhang et al.] 2018). The results on two sets of test data are reported in Table 3 As the SI. ratios increase from 2.4% 9.4%, an improvement of performance of the multi-task U-net is gained, and when it reaches 9.4% SI, the multi-task framework achieves comparable score with the fully supervised version. We emphasize that the 9.4% SL and WL are much lower annotation cost than that of the 100% SL used for fully supervised learning. Example of segmentation results are displayed in Figure]
CONCLUSION
In this paper, we develop multi-task learning framework for image segmentation problems, which relaxes the requirement for numerous and accurate annotations to train the network. is suitable for segmentation problem with dense population of object instances. The model separates the segmentation problem into three smaller tasks. One of them is dedicated to the instance detection and therefore do need exact boundary information. This gives potential flexibility one could concentrate on the classification and rough location of the instances during data collection. The second one focuses on the separation of objects sharing common boundary. The final task aims extracting mixel-wise boundary information. Thanks to the information shared within the multi-task learning, this accurate segmentation can be obtained using very few annotated data.
Our model learns directly the statistics of WL as auxiliary tasks, so no further processing steps needed before training the network. For the partial masks that ignore boundary pixels, the annotation can also be done when the boundaries of object are hard to detect. As small amount of SL is needed and the collection of WL can be fast and cheap, the proposed framework is potentially effective for applications with growing datasets. The weakly annotated set for detection purpose could be augmented necessary and the new images could easily be incorporated into our end-to-end framework.
10

---

Under review as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12)224112445, 2017.
Christoph Baur, Shadi Albarqouni, and Nassir Navab. Semi-supervised deep learning for fully convolutional networks. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 311-319. Springer, 2017.
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In European conference on computer vision, pp. 549-565. Springer, 2016.
Benjamin Bischke, Patrick Helber, Joachim Folz, Damian Borth, and Andreas Dengel. Multi- task learning for segmentation of building footprints with deep neural networks, arXiv preprint arXiv:1709.05932. 2017.
Vicent Caselles, Ron Kimmel, and Guillermo Sapiro. Geodesic active contours. International journal of computer vision, 22(1):61-79, 1997.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint 4rXiv:1412.7062, 2014.
Liang-Chie Ching-Chiihha Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-136, 2015.
Matvey Ezhov, Adel Zakirov, and Maxim Gusarev. Coarse-to-fine volumetric segmentation of teeth in cone-beam ct. arXiv preprint 18Xiv: 10293, 2018.
Arthita Ghosh, Max Ehrlich, Sohil Shah, Larry Davis, and Rama Chellappa. Stacked u-nets for ground material segmentation remote sensing imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 257-261, 2018.
Fidel A Guerrero-Pena. Pedro Marrero Fernandez, Tsang Ing Ren, Mary Yui, Ellen Rothenberg, and Alexandre Cunha. Multiclass weighted loss for instance segmentation of cluttered cells. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 2451-2455. IEEE, 2018.
Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled deep neural network for semi- supervised semantic segmentation. In Advances in neural information processing systems, pp. 1495-1503, 2015.
Zilong Huang, Xinggang Wang. Jiasi Wang, Wenyu Liu, and Jingdong Wang. Weakly-supervine semantic segmentation network with deep seeded region growing. Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 7014-7023, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 15020031., 2015.

---

Under review as a conference paper at ICLR 2020
Longlong Jing, Yucheng Chen, and Yingli Tian. Coarse-to to-fine semantic segmentation from image- level labels. arXiv preprint arXiv:/812.10883 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattemn Recognition, pp. 7482-7491, 2018.
Anna Khoreva, Rodrigo Benenson, Jan Hosang. Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 876-885, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv: 14/2.6980, 2014.
Alexander Kolesnikov and Christoph H Lampert. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In European Conference on Computer Vision, pp. 695-711. Springer, 2016.
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. arXiv preprint arXiv:1902.10421, 2019.
Di Lin. Jifeng Dai. Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble convolu- tional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3159-3167, 2016.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42:60-88 88, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440 2015.
Pawel Mlynarski, Hervé Delingette, Antonio Criminisi, and Nicholas Ayache. Deep learning with mixed supervision for brain tumor segmentation, arXiv preprint arXiv:18/2.04577 2018
George Papandreou, Liang- Chigh Chen, Kevin P Murphy, and Alan L Yuille. Weakly-and semi- supervised learning deep convolutional network for semantic image segmentation. In Proceed- ings of the IEEE international conference computer vision, pp. 1742-1750, 2015.
Christian Perone and Julien Cohen-Adad. Deep semi-supervised segmentation with weight-averaged consistency targets. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 12-19. Springer, 2018
Clément Playout, Renaud Duval, and Farida Cheriet. A multitask learning architecture for simulta- neous segmentation of bright and red lesions fundus images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 101-108. Springer, 2018.
Clément Playout, Renaud Duval, and Farida Cheriet. A novel weakly supervised multitask architecture for retinal lesions segmentation fundus images. IEEE transactions medical imaging, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction
using iterated graph cuts. In ACM transactions on graphics (TOG), volume 23, pp. 309-314, ACM, 2004.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv: (0dv:/706.0000 05098, 2017.
12

---

Under review as a conference paper at ICLR 2020
Meet P Shah, SN Merchant, and Suyash P Awate. Ms-net: Mixed- supervision fully-convolutional net- works for full-resolution segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 377-387. Springer, 2018.
Seung Yeon Shin, Soochahn Lee, II Dong Yun, Sun Mi Kim, and Kyoung Mu Lee. Joint weakly and semi-supervised deep learning for localization and classification of masses in breast ultrasound images. IEEE transactions on medical imaging, 38(3)762-777,, 2019.
Korsuk Sirinukunwattana, Josien PW Pluim, Hao Chen, Xiaojuan Qi, Pheng-Ann Heng, Yun Bo Guo, Li Yang Wang, Bogdan J Matuszewski, Elia Bruni, Urko Sanchez, et al. Gland segmentation in colon histology images: The glas challenge contest. Medical image analysis, 35:489-500 2017.
Fengdong Sun and Wenhui Li. Saliency guided deep network for weakly-supervised image segmen- tation. Pattern Recognition Letters, 120:62-68, 68, 2019.
Tao Sun, Zehui Chen, Wenxiang Yang, and Yin Wang. Stacked u-nets with multi-output for road extraction. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 187-1874. IEEE, 2018.
Satoshi Tsutsui, Tommi Kerola, Shunta Saito, and David J Crandall. Minimizing supervision for free-space segmentation. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 988-997, 2018.
Chengjia Wang, Tom MacGillivray, Gillian Macnaught, Guang Yang, and David Newby. A two- stage 3d unet framework for multi-class segmentation on full resolution image. arXiv preprint arXiv:1804.04533 04341,, 2018a.
Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian Sun, and Chunhua Shen. Repulsion loss: Detecting pedestrians in a crowd. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7774-7783 2018b.
Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, Ming-Ming Cheng, Jiashi Feng, Yao Zhao, and Shuicheng Yan. Ste: A simple to complex framework for seaelyesupervvied semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(11)2231--2200 2017.
Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang. Revisiting dilated convolution: A simple approach for weakly-and semi- supervisd semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1268-7277, 2018.
Jiawei Zhang, Yuzhen Jin, Jilan Xu, Xiaowei Xu, and Yanchun Zhang. Mdu-net: Multi-scale densely connected u-net for biomedical image segmentation. arXiv preprint arXiv: 10XX:0/8//2003322. 201 8.
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes. Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference computer vision, pp. 1529-1537, 2015.
Juan Zhou, Yung Luo, Qi Dou, Hao Chen, Cheng Chen, Gong-Jie Li, Ze-Fei Jiang, and Pheng- Ann Heng. Weakly supervised 3d deep learning for breast cancer classification and localization of the lesions in mr images. Journal of Magnetic Resonance Imaging, 2019.
A APPENDIX
A.1 THE ALGORITHM
We summarize the overall procedures, including the steps of creating the labels and the multi-task training into Algorithm.
13

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Multi-task learning for segmentation with lazy labels
procedure LAZY LABELS(Z)
Choose Z1, I2, Is C Z. [a] can relatively small.
Input: set images I.
2: Select inner regions for each object in T for the detection task
3: Indicate scribbles images of Zz for the separation task
4: Generate few pixel-wise labels from In using interactive segmentation tools (e.g., Grabcut)
return I1, T2, Is and labels
1: procedure MULTI-TASS TASK U-NET TRAINING(L+. (k) a...)
8 denote the labels
Input: labelled sets Zz, loss function weights as for le 1,2,3, Adam parameters r, mini-batch size m. Set the 1st and 2nd momentum vectors m, t as zeros.
3: Initialize the multi-task U-net parameter 0.
4: while Termination criterion is not satisfied do
5: Obtain mini-batch (I,, (()) ).... (Im, (11)
s
(k) can be unknown for some (i, k).
6:
Compute the gradient g + Vo E E an azlz, (4) log s 11; (0, m. v) + Adam, (0, m, v.
7:
b Adam, (-) is Adam iteration
8:
return 0
A.2 FURTHER RESULTS FOR THE ICE CREAM IMAGE DATASET
Figure 8 shows the Dice scores on the 12 test images from Figure 9 One reason for the low accuracy of the single task U-net weak (inaccurate) annotations is that the U-net is not trained recover object boundaries. Therefore, the corresponding dice score on every test image is low, as shown by the blue curve. On the other hand, the network trained performance on only few test images, as shown by the red line. Learning from both strong and weak annotations, the multi-task U-net improves from the other two, and the dice scores are between 0.9 to 0.95 for all of the images.
strong annotations has very good
0.9
U-not wenk amnoiations
U-nse strong annotattion Pt.
Multittook U-nst
0.5
2
10
12
Figure Overall dice scores for the air bubbles and ice crystals plotted versus the test image number. We show the scores for the U-net baseline with weak annotations, strong annotations and pseudo labels (PL) respectively, compared along with the multi-task U-net
A.2.1 ADDITIONAL RESULTS
We compare the segmentation results given by the multi-task U-net with the ones obtained using GAC CACCClselles al. (1997) and Grabcut Rother et al. (2004) (Figure To apply GAC and Grabcut, convert the problem into single class and single object segmentation problem. We use the rough inner regions from the first step the labelling to estimate bounding box around each object. We then apply GAC and Grabcut inside each box. The box-wiss results are aggregated to get the segmentation mask of the whole image.
Both GAC and Grabcut fail to capture the boundaries in various situations, such intensity variations within the objects and background fragments connected to the objects. This demonstrates that a precise labelling requires many additional user interventions and thus the interest of our lazy labelling.
14

---

Under review as a conference paper at ICLR 2020
Image 1
Image 2
Image 3
Image 4
Image 5
Image 6
Image 7
Image 8
Image 9
Image 10
Image 11
Image 12
Figure 9: Twelve patches from the twelve test images respectively.
As shown the third column of Figure 10). the Grabcut repeatedly misclassified small pieces of background parts of the object especially when two objects are close to each other with some weak boundaries. The PL learning methods also give this kind of error in the inference phase. This means that the error has been learned by the segmentation network. The manual detection and correction of such small misclassified regions is tedious task as well, and therefore a limitation of the PI. approach.
A.3 THE H&E-STAINED IMAGE DATESET FOR GLAND SEGMENTATION
Figure 7 shows examples of WL and SL that are used for the segmentation of gland.
15

---

Under review as a conference paper at ICLR 2020
Raw images
GAC
Grabcut
multi-task U-net
Groundtruth
Figure 10: Comparison of multi-task U-net with the GAC and Grabcut methods using the weak labels of the validation data. Green curves: contour the ice crystals. Red curves: air bubbles.
Gland segmentation dataset with SL the first row (highlighted in red), WL in the second row for detection (in orange) and weak labels in the third row for separation (in dark green). The labels for separation can be sparse with some images (see the third row) having zero annotation as the instances are not touching each other.
Figure 11:
16


Reviews:
[{"review_id": "Hyg3vLW9Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission presents a neural network for multi-task learning on sets of labeled data that are largely weakly supervised (in this case, partially segmented instances), augmented by comparatively fewer fully supervised annotations.\nThe multiple tasks are designed to make use of both the weak as well as as full (‘strong’) labels, such that performance on fully annotated machine-generated output is improved.\n\nAs noted in the related work section (Section 2), multi-task methods aim to use benefits from underlying common information that may be ignored in a single-task setting. The network presented here is quite similar to most of these multi-task approaches: a common feature encoder, and partially distinct feature decoding and classification parts.\nThe (minor) novelty mainly comes from the distinct types of weak/strong annotation data fed here: instance scribbles, boundary scribbles, and (some or few) full segmentations. \n\nThe submission is overall well written and provides sufficient clarity and a good overview of the approach.\nSection 3 presents a probabilistic decomposition of the proposed architecture. With some fairly standard assumptions and simplifications, the loss in Eq. 3 becomes rather straightforward (weighted cross entropy)\nThe actual network architecture described in Section 3.2 takes a standard U-Net as a starting point and modifies it in a fairly targeted way for the different expected types of annotations. These annotations (Section 3.3) are cheaper than full labels on a same-size dataset; it is not completely clear, however, if the mentioned scribbles need to capture each instance in the training set, or if some can also be left out. Without this being explicitly mentioned, I will assume the former.\n\nThe experimental evaluation is done reasonably well, although I am not familiar with any of the presented data sets. The SES set seems to be specific to the submission, while the H&E data set has been used at least one other relevant publication (Zhang et al.). My main issue here is that at least on the SES set, which does not seem to be that large, the score difference is not that big, so dataset bias could play some part (which is unproven, but so is the opposite).\nExperimental evaluation does not leave the low-number-of-classes regime, and I’m left wondering how the method might compare on a semantically much richer data set, e.g. Cityscapes. Finally, unmodified U-Net is by now a rather venerable baseline, so I’m also wondering how the proposed multi-task learning could be used in other (more recent) architectures, i.e. whether the idea can be generalized sufficiently.\n\nWhile I think the ideas per se have relatively minor novelty, the combination seems novel to me, and that might warrant publication.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkgaxjChYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for semantic segmentation using \"lazy\" segmentation labels. Lazy labels are defined as coarse labels of the segmented objects. The proposed method is a UNET trained in a multitask fashion whit 3 tasks: object detection, object separation, and object segmentation. The method is trained on 2 datasets: air bubbles, and ice crystals. The proposed method performs better than the same method using only the weakly supervised labels and the one that only uses the sparse labels.\n\nThe novelty of the method is very limited. It is a multitask UNET. The method is compared with one method using pseudo labels. However, this method is not SOTA. Authors should compare with current methods such as:\n - Where are the Masks: Instance Segmentation with Image-level Supervision\n - Instance Segmentation with Point Supervision\n - Object counting and instance segmentation with image-level supervision\n - Weakly supervised instance segmentation using class peak response\n - Soft proposal networks for weakly supervised object localization\n - Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation\nThese methods can use much less supervision (point-level, count-level or image-level) and may work even better.\n\nThe method should be compared on standard and challenging datasets like Cityscapes, PASCAL VOC 2012, COCO, KITTI...\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g4ENhV9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper addresses the problem of learning segmentation models in the presence of weak and strong labels of different types. This is an important problem that arises while learning in data-scarce settings. The presented approach optimizes jointly over the various types of labels by treating each one as a task. They extend the U-net architecture to incorporate the different tasks.\n\nPrior work:\nThere has been other work on incorporating multi-resolution or different types of labels. Here is one that can be cited:\nLabel super-resolution networks (https://openreview.net/forum?id=rkxwShA9Ym)\n\nMajor comments:\n- The motivation for the specific structure of the multi-task blocks is not clear\n- The object boundaries labels can be noisy (i.e s(2) can have noise). How does model deal with this?\n- Is it the case that every image in I_3 is completely labeled - i.e all segments/classes marked?\n- The assumption that s(3) is independent of s(1) and s(2) is not true. Instead of constraining the model to learn masks that respect the various types of labels, it seems they learn from each source independently. It is not clear how the sharing of parameters in the multitask block helps.\n- Can they comment on the applicability of the prior work suggested above?\n\nMinor comments:\n- How do the rough labeling tools work on biomedical data where the objects are more heterogenous patterns where different labels can have very different distribution of pixels. How well will their method generalize in such settings?\n- Can this work be used for segmentation and prediction on crop data?\n\nResults:\n- It seems as if the improvement over the PL baseline (pseudo labels) is incremental? Can the authors provide error bars so the reader knows what the significance of the results is?\n- Can they give a more thorough comparison in terms of human effort? It is interesting to note that only 2 images give 0.82. Would 3 images give 0.94? They need to show the trade-off between additional effort vs gains in performance.\n- What is the performance of MT U-net without the SL images (i.e without task-3)? Table-2 does give some intuition, but authors should add another row with multitask WL\n- Table-3: How well does MDUnet do with 9.4% SL data?\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an architecture for semantic instance segmentation learnable from coarse annotations and evaluates it on two microscopy image datasets, demonstrating its advantage over baseline. While the reviewers appreciate the details of the architecture, they note the lack of evaluation on any of popular datasets and the lack of comparisons with baselines that would be more close to state-of-the-art. The authors do not address this criticism convincingly. It is not clear, why e.g. the Cityscapes or VOC Pascal datasets, which both have reasonably accurate annotations, cannot be used for the validation of the idea. If the focus is on the precision of the result near the boundaries, then one can always report the error near boundaries (this is a standard thing to do). Note that the performance of the baseline models is far from saturated near boundaries (i.e. the errors are larger than mistakes of annotation).

At this stage, the paper lacks convincing evaluation and comparison with prior art. Given that this is first and foremost application paper, lacking some very novel ideas (as pointed out by e.g. Rev1), better evaluation is needed for acceptance.
Decision: Reject