# Part-Based Obstacle Detection Using a Multiple Output Neural Network
## 5 Results and Evaluation
We evaluated the instance segmentation on the CityScapes dataset and compared the results with an R-CNN and Yolact, tested on the same test data from the dataset. The results are presented in Table 2. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Solution** & **Stereo Dataset (IoU)** & **CityScapes Dataset (IoU)** & **KITTI Dataset (IoU)** \\ \hline MONet V1 with YOLO head [39] & 0.64 & 0.63 & - \\ MONNet V1 [39] & 0.74 & 0.80 & 0.55 \\ Proposed system & 0.74 & 0.83 & 0.70 \\ Mask R-CNN [29] & 0.74 & 0.84 & 0.68 \\ DarkNet with YOLO V3 [25] & 0.56 & 0.57 & 0.88 \\ Yolact [31] & 0.73 & 0.84 & 0.68 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Bounding box evaluation. We also present some of the qualitative results of our proposed network, versus Yolact and R-CNN, in Figure 16. The Mask R-CNN would extract well-defined obstacle edges that were more refined, but, in some cases, this method might miss objects (as can be seen in the second row of Figure 16). Yolact predictions are sometimes missing some objects from the scene, as seen in the second or third row of Figure 16; the Yolact solution also seems to predict false positives (fourth row of Figure 16). The popular Mask R-CNN network has a total of 64 million parameters, while Yolact has 50 million parameters; Yolo V3, with a Darknet backbone, features 41 million parameters, while our U-Net- and ResNet-based model features 32 million parameters in total for the three output heads (semantic segmentation, va \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Solution** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline Mask R-CNN [29] & 0.91 & 0.93 & 0.24 & 0.34 \\ Yolact [31] & 0.87 & 0.67 & 0.17 & 0.24 \\ Proposed system & 0.93 & 0.94 & 0.44 & 0.57 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Instance segmentation evaluation. Figure 16: Comparison between Mask R-CNN, Yolact, and our method: the first column represents the input image, the second column shows the ground truth instances, the third column shows the Mask R-CNN results, the fourth column shows the Yolact results, and the fifth column represents our results. quarters). The total prediction time for all three outputs was 0.043 s on average, whereas for the Mask R-CNN, the prediction time was 0.23 s and, for Yolo V3, it was 0.052 s. The Yolact network predicted the output in 0.041 s. If we removed the vanishing point output head, the network was reduced to 24 million parameters and the prediction time was then 0.034 s. If we further removed the semantic segmentation output head, we ended up with a 16-million-parameter network with a prediction time of 0.026 s.  
All the results are presented in Table 3 and the tests were performed on a desktop system, equipped with an Intel i7 CPU and two Nvidia 1080 Ti GPU graphic boards that were used for both training and prediction (evaluation). The prediction time for all three output heads was similar to the one from Yolo V3, which only features obstacle detection (0.057 vs. 0.052 s, on average, on the same input test images). The total computational time for our system was higher, due to the extra post-processing required for the obstacle quarter grouping, labeling, and, finally, bounding box extraction (0.014 s); however, these steps were executed on  
Figure 17: The first column represents the input image, while the second column is the quarter prediction (with the CNN output merged), the third column represents the labeling output, and the fourth column is the resulting bounding boxes of the individual obstacles. The third output head represents the vanishing-point prediction. The vanishing-point coordinates are extracted from the CNN-predicted mask, which takes an additional 0.09 ms on average. The results are presented in Table 4, as already published in our previous work [39]. The evaluation presented in Table 4 was performed using the CityScapes test set and the dataset from [33]. The metric used was NormDist, which represents the RMSE pixel error, divided by the image diagonal.