# Automating Visual Blockage Classification of Culverts with Deep Learning
## 4 Experimental Setup and Evaluation Measures  
Experiments were planned to investigate the performance of existing CNN models for binary classification of culvert images as blocked or clear. Pre-trained CNN models with ImageNet weights were used for this investigation and implemented using Keras with Tensorflow at the backend. Images of dimension \(224\times 224\times 3\) were used as input to model except for NASNet where \(331\times 331\times 3\) and InceptionResNetV2 where \(229\times 229\times 3\) was used. Data augmentations techniques including samplewise standard deviation normalization, horizontal flip, vertical flip, rotation, width shift, and height shift were used in the simulations for improved performance. All the models were tuned with a dropout of 0.2, ReLU activation, and batch normalization. Stochastic gradient descent (SGD) optimizer with a constant learning rate of 0.01 and categorical entropy loss was used. Each model was trained for 30 epochs. For this investigation, the test dataset consisted of selected real images from ICOB (i.e., 91 from blocked, 98 from clear). The rest of the dataset was divided using conventional train:val split with an 80:20 ratio. In addition to the conventional train:val:test dataset split, the 5-fold cross-validation approach has also been implemented and compared towards providing better insight into the performance of deep learning models. The idea of classifying deep visual features extracted from CNN models using conventional machine learning approaches was also implemented. The simulations were performed using Nvidia GeForce RTX 2060 GPU with 6 GB memory and 14 Gbps memory speed. Models were trained at full precision using floating point (FP-32) optimization. The performance of the models was measured in terms of their test accuracy, test loss, precision score, recall score, F1 score, Jaccard Index, ROC curves, and processing times. Each of the evaluation metrics is defined briefly as follows. * **Loss:** Loss is the simplest of the measure to evaluate model training and testing performance. It is the measure of how much instances are classified incorrectly and is the ratio of number of incorrect predictions over total predictions. Minimum value of loss indicated better performance;
* **Accuracy:** In contrast to loss, accuracy is the measure that how much percentage of data instances are classified correctly. It is the ratio of number of correct predictions over total predictions. High value of accuracy represents better performance;
* **Precision Score:** Precision measures the ability of a model to not to classify a negative instance as positive. It answers the question that from all the positive predicted  
Figure 3: Sample instances of clear (First row) and blocked (Second row) culverts from SIC. instances by model, how many were actually positive. The equation below presents the expression for the precision score:  
\[\text{Precision Score}=\frac{\text{True Positive (TP)}}{\text{True Positive (TP) + False Positive (FP)}}\]
* **Recall Score:** Recall answers the question that from all the positive instances, how many were correctly classified by the model. Expression for recall score is given as follows: \[\text{Recall Score}=\frac{\text{True Positive (TP)}}{\text{True Positive (TP) + False Negative (FN)}}\]
* **F1 Score:** F1 score is the single measure which combines both precision and recall by harmonic mean and range between 0 and 1. Higher F1 score indicates the better performance of model. Expression for F1 score is given as follows: \[\text{F1 Score}=2\times\frac{\text{Precision }\times\text{Recall}}{\text{Precision + Recall}}\]
* **Jaccard Index:** In context of classification, Jaccard similarity index score measures the similarity between predicted labels and actual labels. Mathematically, let \(\hat{y}\) denotes the predicted label and \(y\) denotes the actual label, then J index can be expressed as follows. Higher J index indicates better performance of model.