# Implementation of an Obstacle Recognition System for the Blind
## 5 Implementation Results and Performance Evaluation
### Performance Evaluation  
This section compares the obstacle recognition models we used and presents their performance evaluations. First, we retrained the two pre-trained models (SSD MobileNet and Faster R-CNN), using the same training data for each of them. Table 2 shows the performance results of the two models when using the same verification images. The speed of the models and the COCO mean average precision (mAP) are both explained on the Google Object Detection site [12]. It stated that the speed of the Faster R-CNN was greater than that of the SSD MobileNet model. Moreover, the COCO mAP was smaller. For the verification image, images taken from different locations were used. A total of 1000 images were used and they consisted of 500 ballards and 500 crosswalk lights. The SSD MobileNet could be used for minor computing power, but the accuracy of detecting obstacles was only 79.6%. The Faster R-CNN, in contrast, had a higher recognition rate, accurately recognizing 95.1% of obstacles.  
Table 3 shows the accuracy measurement for the crosswalk safety guidance. The data used in this test were taken during two time periods (daytime and nighttime). In addition, the images were classified into three categories: flashing, green, and red. We photographed them for five seconds each. The algorithm confirmed whether the client could safely cross the crosswalk or not. In Table 3, more than 80% of the images distinguished between flashing, green, and red lights during the daytime. However, the algorithm that determined whether a client could safely cross the crosswalk had a significantly lower performance rate during the nighttime, which could be due to the light blur effect caused by the lighting.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Speed (ms)**} & **Common Objects in Context** & \multirow{2}{*}{**Accuracy**} \\  & & **Mean Average Precision** & \\  & & **(COCO mAP)** & \\ \hline Single shot multibox detector (SSD) MobileNet & 30 & 21 & 79.6\% \\ Faster region-convolutional neural network (R-CNN) & 106 & 32 & 95.1\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of obstacle recognition models.  
Figure 6: Display screens recognizing obstacles utilizing the proposed model: (**a**) ballard detection; (**b**) crosswalk light detection.  
Predicting and monitoring the state of a crosswalk light is very important, because any incorrect guidance could impair the safety of the client and cause harm. Therefore, we reanalyzed the low guidance results listed in Table 3. When the crosswalk light was red, guidance for crossing the crosswalk was 0%. This indicated that the probability of guidance results that posed a great risk to the client were close to 0%. However, we are still in the process of developing a filtering algorithm because we know that the system must have higher accuracy for the safest crosswalk guidance possible.  
Table 4 shows the accuracy measurements of the bollard location finding algorithm when determining the position of obstacles when a bollard is recognized. These results were measured using validation images in which one bollard was recognized. The data consisted of images from both day and night. In both cases, when a bollard was recognized in an image, the system could guide the client by informing the client about the location of the bollard.