A multi-task U-net for segmentation with lazy labels

1 INTRODUCTION . Image segmentation has been an active research field in the past decades . Deep learning approaches play an increasingly important role and have become state-of-the-art in various segmentation tasks ( Huang et al. , 2018 ; Khoreva et al. , 2017 ; Tsutsui et al. , 2018 ; Ghosh et al. , 2018 ; Litjens et al. , 2017 ) . Though fully supervised segmentation neural networks have a shown great success , one of their most challenging issues is the need for pixel-level annotations to train them . Obtaining such annotations usually requires a great amount of manual work and is therefore expensive . In this paper , we propose a multi-class and multi-instance segmentation approach that we split into three relevant tasks : detection , separation and segmentation ( cf . Figure 1 ) . Doing so , we obtain a semi-supervised learning approach that is trained with so-called `` lazy '' labels , that is a lot of coarse annotations of class instances together with only a few pixel-wise annotated images that can be obtained from the coarse labels in a semi-automated way . In the following , we will refer to weak ( resp . strong ) annotations for coarse ( resp . accurate ) labels and denote them as WL and SL . Task 1 detects and classifies each object and roughly determines its region through an undersegmentation mask . Instance counting can be obtained as a by-product of this task . As the main objective is instance detection , exact labels for the whole object or its boundary are not necessary at this stage . We use instead weakly annotated images in which a rough region inside each object is marked , cf . the most top left part of Figure 1 . For segmentation problems with a dense population of instances , such as the food components ( see e.g. , Figure 1 ) , cells ( Guerrero-Pena et al. , 2018 ; Ronneberger et al. , 2015 ) , glandular tissue , or people in a crowd ( Wang et al. , 2018b ) , separating objects sharing a common boundary is a well known challenge . We can optionally perform a second task ( Task 2 ) that focuses on the separation of instances that are connected without a clear boundary dividing them . Also for this task we rely on WL to reduce the burden of manual annotations : touching interfaces are specified with rough scribbles , cf . top left part of Figure 1 . Task 3 finally tackles the pixel-wise classification of the instances . It requires strong annotations that are accurate up to the boundaries of the objects . Thanks to the information brought by the weak annotations , we here just need a very small set of accurate segmentation masks , cf . bottom left part of Figure 1 . To that end , we propose to refine some of the coarse labels resulting from task 1 using a semi-automatic segmentation method which requires additional manual intervention . The three tasks are handled by a single deep neural network and are jointly optimized . The network architecture is inspired by the widely used segmentation network named U-net ( Ronneberger et al. , 2015 ) . With all three tasks sharing the same contracting path , we introduce a new multi-task block for the expansive path . The network has three outputs and is fed with a combination of WL and SL described above . Since weakly and strongly annotated training data is shared between the tasks , part of the annotations are missing , especially for task 3 . To accommodate for this we introduce a weighted loss function over the samples . Accurate segmentation labels for training are usually not easy to obtain , however , with our approach we demonstrate that exact labels for the whole training set are not needed for good segmentation learning performance . We evaluate the performance of the proposed approach on two applications , namely for the segmentation of SEM images of food microstructure and stained histology images of glandular tissues . In summary , our contributions are as follows . ( 1 ) . We propose a decomposition of the segmentation problems into three tasks and a corresponding user friendly labeling strategy . ( 2 ) . We develop a multi-task learning framework that learns directly from the manual labels and is trained end-to-end . Our approach outperforms state-of-the-art weakly supervised methods such as Khoreva et al . ( 2017 ) ( 3 ) . The network predicts segmentation mask as well as the object inner regions and touching object interfaces . Touching objects are effectively disconnected as a side product . 2 RELATED WORK . In image segmentation problems , one needs to classify an image at pixel level . It is a vast topic with a diversity of algorithms being developed , including traditional unsupervised methods like k-means that splits the image into homogeneous regions according to image low level features , curve evolution based methods like snakes ( Caselles et al. , 1997 ) , graph-cut based methods like Grabcut ( Rother et al. , 2004 ) , just to name a few . Interactive approaches such as snakes or Grabcut enable getting involved users ’ knowledge by means of initializing regions or putting constraints on the segmentation results . Deep convolutional neural network ( DCNN ) approaches have been developed for segmenting complex images , especially in the semantic setting . In particular , fully convolutional networks ( FCN ) ( Long et al. , 2015 ) replace the last few fully connected layers of a conventional classification network by up-sampling layers and convolutional layers , to preserve spatial information . FCNs have many variants for semantic segmentation . The DeepLab ( Chen et al. , 2018 ) uses a technique called atrous convolution to handle spatial information together with a fully connected conditional random field ( CRF ) ( Chen et al. , 2014 ) for refining the segmentation results . Fully connected CRF can be used as post-processing or can be integrated into the network architecture , allowing for end-to-end training ( Zheng et al. , 2015 ) . Other types of FCNs are deep encoder-decoder networks ( Badrinarayanan et al. , 2017 ; Ronneberger et al. , 2015 ) . They have multiple up-sampling layers for better localizing boundary details . One of the most well-known models is the U-net ( Ronneberger et al. , 2015 ) . It is a fully convolutional network made of a contracting path , which brings the input images into very low resolution features with a sequence of down-sampling , and an expansive path that has an equal amount of up-sampling layers . The higher resolution feature maps on the contracting path are merged with up-sampled layers via long skip connections to recover boundary information after down-sampling . Weakly/semi-supervised segmentation learning . Motivated by the heavy cost of pixel-level annotation needed for fully supervised learning , there has been a growing interest in weakly supervised learning with coarse annotations for semantic segmentation in the last years . Common weak annotations include image-level labels ( Huang et al. , 2018 ; Papandreou et al. , 2015 ; Lee et al. , 2019 ) , bounding boxes ( Khoreva et al. , 2017 ) , scribbles ( Lin et al. , 2016 ) and points ( Bearman et al. , 2016 ) . Most weakly supervised deep learning methods for segmentation are built on top of a classification network . The training of such networks may be realized using segmentation masks explicitly generated from weak annotations ( Wei et al. , 2017 ; Khoreva et al. , 2017 ; Lee et al. , 2019 ; Wei et al. , 2018 ; Tsutsui et al. , 2018 ) . The segmentation masks can be improved recursively , which involves several rounds of training of the segmentation network ( Wei et al. , 2017 ; Jing et al. , 2018 ; Ezhov et al. , 2018 ) . Other techniques do not use any explicit pseudo segmentation masks as training data , but instead compute a composite loss from other guiding principles . For instance , the SEC method ( Kolesnikov & Lampert , 2016 ) combines localization cues ( Seed ) , classification coherence ( Expand ) and Constrainto-boundary with a fully-connected CRF . Special modules could be designed to further exploit the information of sources , such as the Deep Seeded Region Growing ( Huang et al. , 2018 ) and saliency seeded region growing ( Sun & Li , 2019 ) . Semi-supervised deep learning provides an alternative way to reduce the pixel-level annotation burden ( Baur et al. , 2017 ; Perone & Cohen-Adad , 2018 ) . These methods require only a small amount of strongly annotated data and a large set of unlabelled data . Several training methods that combine weak annotations with a limited set of strong annotations have already been explored ( Papandreou et al. , 2015 ; Wei et al. , 2018 ; Lee et al. , 2019 ) . The pixel-wise labels can be integrated through an additional loss function , and used along with their semi-supervised counterparts . Better weak supervision performance can be obtained by feeding the network with learned object localization maps ( Wei et al. , 2018 ; Lee et al. , 2019 ) . However , the networks are trained with image level labels from large scale datasets like ImageNet ( Deng et al. , 2009 ) or PASCAL VOC 2012 ( Everingham et al. , 2015 ) . The weakly and semi-supervised deep learning methods are also explored for object localization or segmentation on relatively smaller datasets , especially in medical imaging . For these methods , the weak annotations can be of image level ( Mlynarski et al. , 2018 ; Playout et al. , 2019 ; Zhou et al. , 2019 ; Shin et al. , 2019 ) or in forms of bounding boxes ( Shah et al. , 2018 ) . Multi-task learning . Multi-task learning algorithms consider several related tasks to improve the overall performance , taking benefits from the underlying common information that may be ignored by a single task learning . In deep neural based multi-task models , the common information is conveyed by soft or hard parameter sharing ( Ruder , 2017 ) . Various multi-task deep learning methods have been developed for segmentation , for example , the stacked U-net for extracting roads from satellite imagery ( Sun et al. , 2018 ) , the two stage 3D U-net framework for 3D CT or MR data segmentation ( Wang et al. , 2018a ) , encoder decoder networks for depth regression , semantic and instance segmentation ( Kendall et al. , 2018 ) , and for building footprint segmentation ( Bischke et al. , 2017 ) . Compared to these works , our method handles the tasks with a multi-task block at different feature resolutions and is designed upon both strong and weak notations . Our work is more closely related to the multi-task learning methods in ( Playout et al. , 2018 ; 2019 ) for retinal lesions segmentation and ( Mlynarski et al. , 2018 ) for brain tumour segmentation . Three tasks are considered in ( Playout et al. , 2019 ) : red lesion segmentation , bright lesions segmentation and image level lesion detection . The proposed encoder-decoder architecture has one branch in the encoder part and two branches into decoder part specified for the red/bright lesion segmentation and the image level classification respectively . The network is weakly supervised as only image level labels are used in one of the training phases . In ( Mlynarski et al. , 2018 ) , a U-net architecture is used for jointly segmenting and classifying brain tumours . An additional branch takes the last but one layer of a standard segmentation U-net as input , followed by a mean pooling layer , and outputs a score for the classification task . Our network structure , similar to these architectures , shares an encoder part for all tasks . The learning is also supervised by a mixture of strong annotations and weak annotations . However , the proposed method does not include supervision from image level annotations and is more specialized in distinguishing the different object instances and clarifying their boundaries in every single image .

Summary: This paper addresses the problem of learning segmentation models in the presence of weak and strong labels of different types. This is an important problem that arises while learning in data-scarce settings. The presented approach optimizes jointly over the various types of labels by treating each one as a task. They extend the U-net architecture to incorporate the different tasks.