# Automating Visual Blockage Classification of Culverts with Deep Learning
## 5 Results and Discussion
From the FN instances in Figure 9, it was observed that for the cases where the image contained contents with a visual appearance similar to blockage material, the image was classified as blocked. This indicated the existence of background clutter/noise problems for this investigation. Background clutter hypothesis was also verified by the intermediate CNN layers activation and heatmaps as given in Figures 10 and 11, respectively. From the layers intermediate activation, it can be observed that at the initial layer, the model retained almost all the visual information as in the input image. However, as the layers go deeper, the model tends to encode higher-level features, such as borders, lines, and edges. Going further deeper results in activation which are not visually interpretable and possess more information related to the class of the input image. Heatmaps for selected FN cases presented in Figure 11 confirmed the hypothesis of background clutter. It can be observed that in most cases, the focus was more on the background contents rather than the culvert opening. Interestingly, in the case of the box culvert, the reflection of light through the culvert was considered by the model as the background which resulted in false classification. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & **Test Accuracy** & **Test Loss/Log Loss** & \multicolumn{2}{c}{**Precision Score**} & \multicolumn{2}{c}{**Recall Score**} & \multicolumn{2}{c}{**F1 Score**} & \multicolumn{2}{c}{**Jaccard Index**} & \multicolumn{2}{c}{**FLOPs**} \\ \hline  & **Conventional** & **5-fold** & **Conventional** & **5-fold** & **Conventional** & **5-fold** & **Conventional** & **5-fold** & **Conventional** & **5-fold** & **Conventional** & **5-fold** \\ \hline  & & & & & & & & & & & & & \\ DarkNet53 & 0.61 & 0.63 & 1.20 & 1.21 & 0.63 & 0.65 & 0.62 & 0.61 & 0.61 & 0.61 & 0.64 & 0.46 & 14.2 G \\ DenseNet21 & 0.77 & 0.79 & 0.47 & 0.57 & 0.77 & 0.80 & 0.77 & 0.79 & 0.77 & 0.79 & 0.62 & 0.66 & 5.7 G \\ InceptionResNetV2 & 0.79 & 0.77 & 0.88 & 0.65 & 0.79 & 0.78 & 0.80 & 0.78 & 0.77 & 0.66 & 0.64 & 13.3 G \\ InceptionV3 & 0.76 & 0.80 & 0.74 & 0.64 & 0.76 & 0.80 & 0.76 & 0.80 & 0.76 & 0.80 & 0.62 & 0.66 & 5.69 \\ MobileNet & 0.81 & 0.78 & 0.51 & 0.59 & 0.81 & 0.79 & 0.81 & 0.79 & 0.81 & 0.78 & 0.69 & 0.65 & 1.5 G \\ ResNet50 & 0.78 & 0.79 & 0.70 & 0.62 & 0.78 & 0.76 & 0.78 & 0.76 & 0.78 & 0.79 & 0.64 & 0.65 & 7.95 \\ VGG6 & 0.71 & 0.57 & 0.58 & 0.79 & 0.72 & 0.43 & 0.70 & 0.58 & 0.70 & 0.48 & 0.55 & 0.41 & 30.7 G \\ EfficientNet3 & 0.78 & 0.79 & **0.46** & 0.57 & 0.78 & 0.80 & 0.78 & 0.79 & 0.78 & 0.79 & 0.64 & 0.66 & 1.97 G \\ NASNet & **0.84** & **0.85** & 0.58 & **0.58** & **0.85** & **0.85** & **0.84** & **0.85** & **0.84** & **0.85** & **0.73** & **0.73** & 4/8.5 G \\ \hline  & & & & & & & & & & & & & \\ \hline  & & & & & & & & & & & & & & \\ SVM & 0.55 & 0.63 & 15.53 & 12.61 & 0.70 & 0.64 & 0.57 & 0.63 & 0.46 & 0.63 & 0.38 & 0.47 & NA \\ RF & 0.47 & 0.57 & 18.27 & 14.80 & 0.47 & 0.57 & 0.48 & 0.57 & 0.40 & 0.56 & 0.31 & 0.40 & NA \\ xgboost & 0.50 Figure 5: ROC curves for implemented deep learning models. Figure 6: Confusion matrices of implemented CNN models for blockage detection (Conventional Train:Val:Test split). Figure 8: Graphical comparison of implemented CNN models for test performance. Figure 7: Confusion matrices of implemented CNN models for blockage detection (5-fold cross validation). Figure 10: Selected intermediate ResNet50 CNN layer activation. Figure 9: Selected false positive (First row) and false negative (Second row) instances. Given that, existing CNN models, conventional machine learning approaches, and classification of deep CNN visual features using conventional machine learning approaches did not achieve high enough accuracy (i.e., maximum 5-fold accuracy of 85% achieved for NASNet model) that they can be deployed to replace the manual visual inspection of culverts. However, they can potentially be used to partially automate the process of manual visual inspection of culverts.