# Semantic Image Segmentation Using Scant Pixel Annotations
## 5 Experimental Results and Discussion
### Discussion  
The SSPA is a novel approach for generating high-performing deep learning models for image semantic segmentation using scant expert image annotations. Automated methods generating pseudo-labels are integrated with an iterative active learning approach to selectively perform manual annotation and improve model performance. We used an ensemble of MC-WS segmentation modules to generate pseudo-labels. We also considered other popular choices, such as grab-cut [55] to generate pseudo-labels and chose MC-WS based on its relative superior performance. Pseudo-labeling approaches other than MC-WS may perform better for other applications, and these can be easily incorporated into the SSPA approach. Note that using a method that generates high-quality pseudo-labels is beneficial to the SSPA, but it is not essential to its success. In the SSPA approach, the pixel replacement effort required by the expert is inversely proportional to the initial pseudo-label quality. In the worst-case scenario, a low-quality initial pseudo-label set has to be compensated by the extra labeling effort from the experts. In the SSPA, images that need expert attention are chosen based on their model prediction entropy values. We employ entropy as the uncertainty sampling measure for the active learning process, over marginal, ratio, and least confidence sampling techniques. Entropy-based sampling is well known and has been shown to be well suited for selecting candidates for classification and segmentation tasks [56]. In the SSPA, we compute entropy value for each image and use these values to identify the top-k images whose certain pixels have to be manually annotated by experts. A high entropy (HE) value for an image indicates an image where most pixel predictions are uncertain (probability in the range \(0.5\pm\delta\)) in that image. If an image with HE value is selected as one of the top-k images for annotation by experts, then pixels with prediction values around 0.5 are labeled by the experts in order to reduce the uncertainty of predictions. Alternatively, a low entropy (LE) value for an image indicates that most of the pixel predictions are made with high confidence. If an image with LE entropy value is selected as one of the top-k images for annotation by experts, then this means that there are sufficient pixels with uncertain predictions (probability in the range \(0.5\pm\delta\)) in that image, and these need to be labeled by experts to improve the performance of the model. Table 1 illustrates the experiments conducted on both high entropy and low entropy and the best-performed strategy (HE or LE) for each dataset. The uncertainty range threshold \(\delta\) is one of the two parameters to the SSPA that was empirically determined to be 0.05 for our experiments. The \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multirow{2}{*}{**Expert Labels**} & \multicolumn{3}{c}{**Dataset [IOU / Dice] (\%) \(\uparrow\)**} \\ \cline{3-5}  & & **EM** & _Melanoma_ & _Biofilm_ \\ \hline FS: U-Net & \(P\) (100\%) & **82.3 / 90.3** & 76.4 / 96.2 & 60.9 / 75.4 \\ FS: DeepLabV3+ & \(P\) (100\%) & 81.6 / 89.1 & 76.0 / 86.2 & 61.6 / 74.4 \\ \hline WS: Grab-Cut + U-Net & \(I\) (100\%) & 76.1 / 85.2 & 65.5 / 92.7 & 56.4 / 62.0 \\ WS: MC-WS + U-Net & \(I\) (100\%) & 76.3 / 86.5 & 68.3 / 95.5 & **69.7 / 82.0** \\ \hline SSPA + LE & \(P\) (<26\%) & 81.8 / 90.0 & 71.8 / 95.8 & 46.9 / 63.4 \\ SSPA + HE & \(P\) (<5\%) & 77.6 / 87.4 & **82.4 / 97.3** & 69.1 / 81.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Evaluation of the effect of SSPA approach on segmentation quality using both fully supervised and weakly supervised approaches. Best result is shown in bold. For IOU and Dice score metrics, the higher the number, the better (\(\uparrow\)). \(I\) represents the image-level labels and \(P\) represents pixel-level labels; percentages are cumulative expert labels across all the iterations of active learning.