A multi-task U-net for segmentation with lazy labels

1 INTRODUCTION . Image segmentation has been an active research field in the past decades . Deep learning approaches play an increasingly important role and have become state-of-the-art in various segmentation tasks ( Huang et al. , 2018 ; Khoreva et al. , 2017 ; Tsutsui et al. , 2018 ; Ghosh et al. , 2018 ; Litjens et al. , 2017 ) . Though fully supervised segmentation neural networks have a shown great success , one of their most challenging issues is the need for pixel-level annotations to train them . Obtaining such annotations usually requires a great amount of manual work and is therefore expensive . In this paper , we propose a multi-class and multi-instance segmentation approach that we split into three relevant tasks : detection , separation and segmentation ( cf . Figure 1 ) . Doing so , we obtain a semi-supervised learning approach that is trained with so-called `` lazy '' labels , that is a lot of coarse annotations of class instances together with only a few pixel-wise annotated images that can be obtained from the coarse labels in a semi-automated way . In the following , we will refer to weak ( resp . strong ) annotations for coarse ( resp . accurate ) labels and denote them as WL and SL . Task 1 detects and classifies each object and roughly determines its region through an undersegmentation mask . Instance counting can be obtained as a by-product of this task . As the main objective is instance detection , exact labels for the whole object or its boundary are not necessary at this stage . We use instead weakly annotated images in which a rough region inside each object is marked , cf . the most top left part of Figure 1 . For segmentation problems with a dense population of instances , such as the food components ( see e.g. , Figure 1 ) , cells ( Guerrero-Pena et al. , 2018 ; Ronneberger et al. , 2015 ) , glandular tissue , or people in a crowd ( Wang et al. , 2018b ) , separating objects sharing a common boundary is a well known challenge . We can optionally perform a second task ( Task 2 ) that focuses on the separation of instances that are connected without a clear boundary dividing them . Also for this task we rely on WL to reduce the burden of manual annotations : touching interfaces are specified with rough scribbles , cf . top left part of Figure 1 . Task 3 finally tackles the pixel-wise classification of the instances . It requires strong annotations that are accurate up to the boundaries of the objects . Thanks to the information brought by the weak annotations , we here just need a very small set of accurate segmentation masks , cf . bottom left part of Figure 1 . To that end , we propose to refine some of the coarse labels resulting from task 1 using a semi-automatic segmentation method which requires additional manual intervention . The three tasks are handled by a single deep neural network and are jointly optimized . The network architecture is inspired by the widely used segmentation network named U-net ( Ronneberger et al. , 2015 ) . With all three tasks sharing the same contracting path , we introduce a new multi-task block for the expansive path . The network has three outputs and is fed with a combination of WL and SL described above . Since weakly and strongly annotated training data is shared between the tasks , part of the annotations are missing , especially for task 3 . To accommodate for this we introduce a weighted loss function over the samples . Accurate segmentation labels for training are usually not easy to obtain , however , with our approach we demonstrate that exact labels for the whole training set are not needed for good segmentation learning performance . We evaluate the performance of the proposed approach on two applications , namely for the segmentation of SEM images of food microstructure and stained histology images of glandular tissues . In summary , our contributions are as follows . ( 1 ) . We propose a decomposition of the segmentation problems into three tasks and a corresponding user friendly labeling strategy . ( 2 ) . We develop a multi-task learning framework that learns directly from the manual labels and is trained end-to-end . Our approach outperforms state-of-the-art weakly supervised methods such as Khoreva et al . ( 2017 ) ( 3 ) . The network predicts segmentation mask as well as the object inner regions and touching object interfaces . Touching objects are effectively disconnected as a side product . 2 RELATED WORK . In image segmentation problems , one needs to classify an image at pixel level . It is a vast topic with a diversity of algorithms being developed , including traditional unsupervised methods like k-means that splits the image into homogeneous regions according to image low level features , curve evolution based methods like snakes ( Caselles et al. , 1997 ) , graph-cut based methods like Grabcut ( Rother et al. , 2004 ) , just to name a few . Interactive approaches such as snakes or Grabcut enable getting involved users ’ knowledge by means of initializing regions or putting constraints on the segmentation results . Deep convolutional neural network ( DCNN ) approaches have been developed for segmenting complex images , especially in the semantic setting . In particular , fully convolutional networks ( FCN ) ( Long et al. , 2015 ) replace the last few fully connected layers of a conventional classification network by up-sampling layers and convolutional layers , to preserve spatial information . FCNs have many variants for semantic segmentation . The DeepLab ( Chen et al. , 2018 ) uses a technique called atrous convolution to handle spatial information together with a fully connected conditional random field ( CRF ) ( Chen et al. , 2014 ) for refining the segmentation results . Fully connected CRF can be used as post-processing or can be integrated into the network architecture , allowing for end-to-end training ( Zheng et al. , 2015 ) . Other types of FCNs are deep encoder-decoder networks ( Badrinarayanan et al. , 2017 ; Ronneberger et al. , 2015 ) . They have multiple up-sampling layers for better localizing boundary details . One of the most well-known models is the U-net ( Ronneberger et al. , 2015 ) . It is a fully convolutional network made of a contracting path , which brings the input images into very low resolution features with a sequence of down-sampling , and an expansive path that has an equal amount of up-sampling layers . The higher resolution feature maps on the contracting path are merged with up-sampled layers via long skip connections to recover boundary information after down-sampling . Weakly/semi-supervised segmentation learning . Motivated by the heavy cost of pixel-level annotation needed for fully supervised learning , there has been a growing interest in weakly supervised learning with coarse annotations for semantic segmentation in the last years . Common weak annotations include image-level labels ( Huang et al. , 2018 ; Papandreou et al. , 2015 ; Lee et al. , 2019 ) , bounding boxes ( Khoreva et al. , 2017 ) , scribbles ( Lin et al. , 2016 ) and points ( Bearman et al. , 2016 ) . Most weakly supervised deep learning methods for segmentation are built on top of a classification network . The training of such networks may be realized using segmentation masks explicitly generated from weak annotations ( Wei et al. , 2017 ; Khoreva et al. , 2017 ; Lee et al. , 2019 ; Wei et al. , 2018 ; Tsutsui et al. , 2018 ) . The segmentation masks can be improved recursively , which involves several rounds of training of the segmentation network ( Wei et al. , 2017 ; Jing et al. , 2018 ; Ezhov et al. , 2018 ) . Other techniques do not use any explicit pseudo segmentation masks as training data , but instead compute a composite loss from other guiding principles . For instance , the SEC method ( Kolesnikov & Lampert , 2016 ) combines localization cues ( Seed ) , classification coherence ( Expand ) and Constrainto-boundary with a fully-connected CRF . Special modules could be designed to further exploit the information of sources , such as the Deep Seeded Region Growing ( Huang et al. , 2018 ) and saliency seeded region growing ( Sun & Li , 2019 ) . Semi-supervised deep learning provides an alternative way to reduce the pixel-level annotation burden ( Baur et al. , 2017 ; Perone & Cohen-Adad , 2018 ) . These methods require only a small amount of strongly annotated data and a large set of unlabelled data . Several training methods that combine weak annotations with a limited set of strong annotations have already been explored ( Papandreou et al. , 2015 ; Wei et al. , 2018 ; Lee et al. , 2019 ) . The pixel-wise labels can be integrated through an additional loss function , and used along with their semi-supervised counterparts . Better weak supervision performance can be obtained by feeding the network with learned object localization maps ( Wei et al. , 2018 ; Lee et al. , 2019 ) . However , the networks are trained with image level labels from large scale datasets like ImageNet ( Deng et al. , 2009 ) or PASCAL VOC 2012 ( Everingham et al. , 2015 ) . The weakly and semi-supervised deep learning methods are also explored for object localization or segmentation on relatively smaller datasets , especially in medical imaging . For these methods , the weak annotations can be of image level ( Mlynarski et al. , 2018 ; Playout et al. , 2019 ; Zhou et al. , 2019 ; Shin et al. , 2019 ) or in forms of bounding boxes ( Shah et al. , 2018 ) . Multi-task learning . Multi-task learning algorithms consider several related tasks to improve the overall performance , taking benefits from the underlying common information that may be ignored by a single task learning . In deep neural based multi-task models , the common information is conveyed by soft or hard parameter sharing ( Ruder , 2017 ) . Various multi-task deep learning methods have been developed for segmentation , for example , the stacked U-net for extracting roads from satellite imagery ( Sun et al. , 2018 ) , the two stage 3D U-net framework for 3D CT or MR data segmentation ( Wang et al. , 2018a ) , encoder decoder networks for depth regression , semantic and instance segmentation ( Kendall et al. , 2018 ) , and for building footprint segmentation ( Bischke et al. , 2017 ) . Compared to these works , our method handles the tasks with a multi-task block at different feature resolutions and is designed upon both strong and weak notations . Our work is more closely related to the multi-task learning methods in ( Playout et al. , 2018 ; 2019 ) for retinal lesions segmentation and ( Mlynarski et al. , 2018 ) for brain tumour segmentation . Three tasks are considered in ( Playout et al. , 2019 ) : red lesion segmentation , bright lesions segmentation and image level lesion detection . The proposed encoder-decoder architecture has one branch in the encoder part and two branches into decoder part specified for the red/bright lesion segmentation and the image level classification respectively . The network is weakly supervised as only image level labels are used in one of the training phases . In ( Mlynarski et al. , 2018 ) , a U-net architecture is used for jointly segmenting and classifying brain tumours . An additional branch takes the last but one layer of a standard segmentation U-net as input , followed by a mean pooling layer , and outputs a score for the classification task . Our network structure , similar to these architectures , shares an encoder part for all tasks . The learning is also supervised by a mixture of strong annotations and weak annotations . However , the proposed method does not include supervision from image level annotations and is more specialized in distinguishing the different object instances and clarifying their boundaries in every single image .

Summary: This paper proposes a method for semantic segmentation using "lazy" segmentation labels. Lazy labels are defined as coarse labels of the segmented objects. The proposed method is a UNET trained in a multitask fashion whit 3 tasks: object detection, object separation, and object segmentation. The method is trained on 2 datasets: air bubbles, and ice crystals. The proposed method performs better than the same method using only the weakly supervised labels and the one that only uses the sparse labels.