# Semantic Structure from Motion for Railroad Bridges Using Deep Learning
## 4 Validation Test  
To verify the performance of the proposed SsfM system, an experiment was carried out using the test data collected from the Osong 3 rd test track in Osong-eup, Heungdeok-gu, Cheongju-si, Chungcheongbuk-do, Korea. A total of 183 test data were collected from the Osong 3 rd test track using drones as test data and the semantic segmentation technique developed in this study was applied to automatically classify the components of the bridge. In addition, SfM was applied based on 183 images with semantic information and a 3D point cloud model including information on the components of the bridge was obtained.  
In general, a confusion matrix, as shown in Table 4, was used to evaluate the result of semantic segmentation. Measures to evaluate using the error matrix include accuracy, intersection over union (IoU), precision, recall, and F1 score.  
Accuracy is one of the measures used to intuitively evaluate the performance of a classification model, as shown in Equation (4). However, data with unbalanced results can distort the performance of the model and must be expressed using other methods.  
\[Accuracy=\frac{TP+TN}{TP+FP+FN+TN} \tag{4}\]  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{2}{c}{**Predicted Class**} \\ \cline{3-4}  & & **True** & **False** \\ \hline \multirow{2}{*}{Actual class} & True & True positive (TP) & False negative (FN) \\ \cline{2-4}  & False & False positive (FP) & True negative (TN) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Confusion matrix.  
Figure 11: Principle of the SfM photogrammetry.  
IoU is an intuitive measure for the evaluation of the performance of a classification model as an intersection. It is most often used to evaluate the prediction results in semantic segmentation and object detection,  
\[IoU=\frac{TP}{FN+TP+FP} \tag{5}\]  
Precision and recall are usually used together and exhibit an inverse relationship. However, both measures have drawbacks, so that they are used together in the F1 score. The F1 score is an index representing the harmonic average of precision and recall and can accurately evaluate the performance of the model even with unbalanced data. Precision, recall, and F1 scores (denoted BF) are defined in Equations (6)-(8), respectively.  
\[Precision=\frac{TP}{TP+FP} \tag{6}\]  
\[Recall=\frac{TP}{TP+FN} \tag{7}\]  
\[F1\ score=2\times\frac{Precision\ \times Recall}{Precision\ + Recall} \tag{8}\]  
The proposed semantic segmentation network classified 183 test data collected from the Osong  
After the application of 183 data to the bridge component classification network, 3D PCD were generated by applying SfM. The results are shown in Figure 13 and Table 6. The results of semantic segmentation could be successfully expressed in the 3D point cloud model. The average precision was approximately 74%, the IoU was approximately 65%, and the BF-score was approximately 55%. It was expected that an additional error will occur in SfM, which converts 2D images to a 3D point cloud model, resulting in lower SfM results than the 2D segmentation results. However, as the semantic segmentation results of several images were averaged to a 3D point of one, the IoU and BF-score of a specific class (pier) slightly increased.