# Semantic Image Segmentation Using Scant Pixel Annotations
## 1 Introduction  
Semantic image segmentation is the task of assigning to each pixel the class of its enclosing object or region as its label, thereby creating a segmentation mask. Due to its wide applicability, this task has received extensive attention from experts in several areas, such as autonomous driving, robot navigation, scene understanding, and medical imaging. Owing to its huge success, deep learning has become the de-facto choice for semantic image segmentation. Recent approaches have used convolutional neural networks (CNNs) [1; 2] and fully convolutional networks (FCNs) [3; 4; 5] for this task and achieved promising results. Several recent surveys [6; 7; 8; 9; 10; 11] describe the successes of semantic image segmentation and directions for future research. Typically, large volumes of labeled data are needed to train deep CNNs for image analysis tasks, such as classification, object detection, and semantic image segmentation. This is especially so for semantic image segmentation, where each pixel in each training image has to be labeled or annotated in order to infer the labels of the individual pixels of a given test image. The availability of densely annotated images in sufficient numbers is problematic, particularly in domains such as material science, engineering, and medicine, where annotating images is time consuming and requires significant user expertise. For instance, while reading retinal images to identify unhealthy areas, it is common for graders (with ophthalmology training) to discuss each image at length to carefully resolve several confounding and subtle image attributes [12; 13; 14]. Labeling cells, cell clusters, and microbial byproducts in biofilms take up to two days per image on average [15; 16; 17]. Therefore, it ishighly beneficial to develop high-performance deep segmentation networks that can train with scantly annotated training data. In this paper, we propose a novel approach for semantic segmentation of images that can work with datasets with scant expert annotations. Our approach, segmentation with scant pixel annotations (SSPA) combines active learning and semi-supervised learning approaches to build segmentation models where segmentation masks are generated using automatic pseudo-labeling as well as by using expert manual annotations on a selective small set of images. The proposed SSPA approach employs a marker-based watershed algorithm based on image morphology to automatically generate pseudo-segmentation masks for the full training dataset. The performance of a segmentation model generated using these pseudo-masks is analyzed, and a sample of images to be annotated by experts is selected. These images with expert-generated masks along with images with pseudo-masks are used to train the next model. The process is iterated to successively generate a sequence of models until either the performance improvement plateaus or no further refinements are possible. The approach uses top-k (bottom-k) image entropy over pixel prediction confidence probabilities to identify the sample images at each iteration. Despite the careful selection of image samples in each iteration to limit the overall manual annotation effort, annotating each image in its entirety can be tedious for experts. This is especially so for the high information density of the segmentation task, where each and every pixel needs to be classified. The proposed _SSPA_ approach reduces annotation effort by using pixels as the unit of annotation instead of annotating entire images or image patches. For each image that is selected for manual annotation, only pixels within a specified uncertainty range are marked for expert annotation. For the rest of the image, a set of replacement rules that leverage useful patterns learned by the segmentation model are used to automatically assign labels. The segmentation mask for an image includes these labels along with expert annotations, and is used to generate the next model.