# Failure Detection for Semantic Segmentation on Road Scenes Using Deep Learning
## 3 Experimental Results
### Experimental Results  
#### 3.3.1 Quantitative Results  
Various FCL structures were applied to predict mIoU values and used in our task. According to the various FCL structures, the evaluation indicators are as follows: selecting the most efficient model, mIoU prediction accuracy, number of parameters, and run time. Similar to the image classification in [32], as the number of nodes in the FCL increases, the mIoU prediction accuracy becomes higher. However, real-time detection must be guaranteed. Therefore, 256-20-1, a structure with high accuracy that does not require a run time of 1 s, was selected. The evaluation indices obtained using the selected network structure are shown in Table 4. The mIoU prediction accuracy was calculated using (5) and (6). Experiments were conducted in two directions using the selected network structure:  
* Additional information: Additional information is learned by increasing the number of channels to improve prediction performance. We experimented with the cases \begin{table}
\begin{tabular}{c c c c} \hline \hline
**FCL Structure** & **mIoU Predicition Accuracy (\%)** & **Params (M)** & **Run Time (s)** \\ \hline \(10\times 20\times 1\) & 93.08 & 172.6 & 0.084 \\ \(16\times 20\times 1\) & 93.06 & 270.9 & 0.094 \\ \(32\times 20\times 1\) & 93.17 & 533.1 & 0.122 \\ \(64\times 20\times 1\) & 93.10 & 1057.5 & 0.194 \\ \(128\times 20\times 1\) & 93.01 & 2106.2 & 0.318 \\ \(256\times 20\times 1\) & 93.21 & 4203.5 & 0.558 \\ \(512\times 20\times 1\) & 93.32 & 8398.4 & 1.048 \\ \(1024\times 20\times 1\) & 93.14 & 16,788.1 & 2.02 \\ \(2048\times 20\times 1\) & 93.24 & 33,567.5 & 4.038 \\ \(4096\times 20\times 1\) & 93.25 & 67,126.2 & 7.76 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Mean intersection of union (mIoU) prediction accuracy and run time of various fully connected layer (FCL) structures. of RGB, RGB hue, and RGB segmentation (RGBSeg) to observe the difference when learning additional information. * Real-time application: The image size was reduced and trained to secure real-time performance. In adding one more channel with additional information, the accuracy improved when adding a hue channel corresponding to the color and adding a segmentation map. The increase in run time due to the increased network parameters as the channel was added was insignificant. We decided to use an input image with four RGB channels and a segmentation map for the best results (Table 5). After fixing the input image with RGBSeg with four channels, the input size was adjusted to reduce the network parameters. The experiments were conducted by reducing the existing 1024 \(\times\) 512 input image to two times and four times. As the computation amount was reduced, the run time was similarly reduced with a real-time application, but the accuracy also decreased (Table 6). As expected, when the image size was the largest, the accuracy was highest, and, when the image size was halved, the accuracy decreased by about 1.2%. The detection results of the limit situations according to each input size of the image are listed in Table 7.  
The accuracy of failure detection significantly decreased as the input size decreases. However, as presented in Table 7, the precision, recall, and F1-score values were calculated to determine why the mIoU prediction accuracy was higher than 512 \(\times\) 256. As a result of checking the value, the image size was reduced so much that the network could learn much less, and training was performed to reduce the loss function. Therefore, all testing set images were recognized as failures due to overfitting on most GT mIOU values. In addition, when comparing the results for images of size 1024 \(\times\) 512 and 512 \(\times\) 256, the accuracy difference is almost 15%. Although the run time differed by about 2.7 times, the detection accuracy was significantly lower; therefore, it was not efficient to reduce the input size of the image to provide real-time characteristics. The accuracy of MSE loss function and the accuracy of modified loss function differ as follows (Table 8).