# Pavement Defect Segmentation in Orthoframes with a Pipeline of Three Convolutional Neural Networks
## 11 Conclusions  
Transportation infrastructure systems are essential to the government and commerce. Camera-based pavement distress detection plays a critical role in pavement maintenance and rehabilitation and early detection of distresses developing in pavements offers significant cost savings resulting from timely maintenance and repair. Automated detection of distresses from pavement images is, however, a challenging problem due to a number of factors--variations in image quality, non-uniformity of cracks and surface texture, lack of sufficient background illumination, and so forth--which keeps the research community looking for newer and more efficient methods and algorithms. Recent advances in computer vision, largely thanks to the emergence of deep learning, have led researchers to develop robust tools for analyzing pavement images at improved pavement distress detection accuracy. To address the pavement distress detection problems in Estonia, we have proposed a novel pavement distress detection system that employs three ConvNets and is applied to the orthoframes collected by a mobile mapping system. The first of three networks in the system (RSN) extracts the road area from an orthoframe to be inspected. The second one (DDN) divides the extracted road area into individual patches and labels them into defective and non-defective ones. Third and final network (DSN) segments the defective patches for defects. These networks are expected to operate in a mutually reinforcing manner, however, mistakes that are made by RSN or DDN do propagate through the system. In the end, the defect mask is assembled from segmented patches. The whole process is improved iteratively by active learning. We have utilized a number of metrics to assess the performance of the system: pixel-wise intersection over union (IoU), edge-tolerant intersection over union (eloU), instance-level intersection over union (iloU) and object-level based defect classification precision (cPr) and recall (CRC). Currently, after 5 iterations of active learning, the PDDS has obtained a pixel-wise IoU of 0.515, eloU of 0.581, instance-level intersection over union (iloU) of 0.373, cPr of 0.583 and cRc of 0.641, which have been computed by comparing the 4096 \(\times\) 4096 ground truth defect masks with the 4096 \(\times\) 4096 predicted defect masks over the test set. Visual assessment of the produced output indicates that for the majority of orthoframes, the results are satisfactory, despite the complexity of the task. Each network in the PDDS contributes to the overall performance at a different degree. The analysis shows that road area extraction--easily the most solvable of three subtasks--can be performed with a high IoU above 0.95, whereas recall obtained by the DDN is barely 0.8. We consider recall of the DDN to be more important measure than precision because inability to recognize patches that contain defects means that defects on these patches will never be discovered, much less segmented, \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & **RSN** & **DDN** & **DSN** & Pre- and Postprocessing & **Total** \\ \hline RSN, ResNet34 DDN, ResNet34 DSN & 0.83 s & 2.11 s & 0.63 s & 1.07 s & **4.64 s** \\ RSN, ResNet34 DDN, Inception-v4 DSN & 0.83 s & 2.11 s & 0.84 s & 1.07 s & **4.85 s** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: PDDS inference speed, separated by different modules of the pipeline. For each column, the seconds taken per orthoframe is shown. whereas inclusion of patches that do not contain any defects still leaves the possibility that the DSN can adequately handle these patches. Nevertheless, most of the mistakes made by the system can still be contributed to the DSN as segmentation of defects is by nature the most error-prone of three subtasks and further improvements must be sought in this area: by further iterations of active learning and/or by applying more efficient networks/learning schemes.