# Semantic Image Segmentation Using Scant Pixel Annotations
## 3 The SSPA Algorithm  
The inputs to the SSPA algorithm (Figure 1) are a set of images, \(U=\{u_{1},\ldots,u_{N}\}\) and an optional set of corresponding ground truth, \(GT=\{g_{1},\ldots,g_{M}\}\), binary labels, where \(M\leq N\). The SSPA employs an iterative algorithm that uses a sequence of training sets, \(T=(T_{1},\ldots,T_{k})\) to build a sequence of models, \((M_{1},\ldots,M_{k})\). Model \(M_{i}\) is generated at the \(i^{th}\) iteration using training set \(T_{i}\). Let \(L_{i}\) be the set of pixel-level binary labels for each training sample in \(T_{i}\). Each \(T_{i}\) comprises image-label pairs \(<t_{p},l_{p}>\), where the \(l_{p}\) refers to a training sample from \(U\), and \(l_{p}\) refers to the corresponding pseudo labels from \(L_{i}\), distinct from \(GT\). We apply \(M_{i}\) to each training sample in \(T_{i}\) to obtain a set of confidence predictions \(C_{i}\). For each image \(t_{p}\) in \(T_{i}\), \(C_{i}\) contains an element \(e_{p}\) with a pair of values, \((conf,label)\) for each pixel in \(t_{p}\), where \(conf\) denotes the prediction confidence value that the pixel belongs to class \(1\) and \(label\) denotes the pixel label assigned by \(M_{i}\). Min-max normalization of raw \(conf\) values of all pixels in \(e_{p}\) is used to normalize them to a \([0..1]\) interval to construct \((\overline{conf},label)\) pairs for each \(e_{p}\). We use \(\overline{C_{i}}\) to refer to \(C_{i}\) containing the normalized value, label pairs for each image in \(T_{i}\). The entropy of a sample image is found using its corresponding normalized prediction confidence values. The mean entropy \(\mu_{i}\) of \(T_{i}\) (and of \(M_{i}\)) is calculated as the mean of the entropy of all images in \(T_{i}\). The SSPA algorithm can be divided into three main steps as described below. **Initial Model and Initial Pseudo-label Set Generation:** The marker controlled watershed (MC-WS) algorithm was employed to avoid the over-segmentation caused due to noise and irregularities that typically occur in the use of the watershed transform. The MC-WS floods the topographic image surface from a predefined set of markers, thereby preventing over-segmentation. To apply MC-WS on each image, an approximate estimate of the foreground objects in the image was first found using binarization. White noise and small holes in the image were removed using morphological opening and closing, respectively. To extract the sure foreground region of the image, distance transform was then used to apply a threshold. Then, to extract the sure background region of the image, dilation was applied on the image. The boundaries of the foreground objects were computed as the difference between the sure foreground and sure background regions. Marker labeling was implemented by labeling all sure regions with positive integers and labeling all unknown (or boundary) regions with a 0. Finally, watershed was applied on the maker image to modify the boundary region to obtain the watershed segmentation mask or binary label of the image. Using MC-WS, we created an ensemble of three watershed segmentation modules, \(\{w_{1},ws_{2},ws_{3}\}\) and applied it to the set \(U\) to generate labels, \(\{L_{ws_{1}},L_{ws_{2}},L_{ws_{3}}\}\). Use majority voting to determine the initial set of pseudo binary labels, \(L_{1}\). Train a segmentation network on pair \(<U,L_{1}>\) to obtain initial model \(M_{1}\) (refer to lines 23-27 in Algorithm 1). We use model \(M_{1}\) to generate the prediction confidence values \(C_{1}\) and normalized prediction confidence set \(\overline{C_{1}}\) for \(U\). Let \(\mu_{1}\) be the mean entropy of \(M_{1}\). Figure 1: SSPA architecture. In this figure, \(k\) represents the termination condition described in Algorithm 1.