## 4 Experimental Setup and Results
### Classification between Street Art and Illegal Graffiti
#### 4.1.1 Results from the Classification Model  
Figure 12 presents some cases correctly classified using the trained model. Two metrics were measured and tracked to evaluate the results: accuracy and categorical cross-entropy. The accuracy calculates how often predictions are equal to the labels. The categorical cross-entropy is one of the most commonly used functions for deep learning multi-class classification problems [35] because it computes the cross-entropy loss between the labels and predictions, i.e., it measures the difference between the two probability distributions (predicted and actual).  
Figure 13 shows the accuracy and categorical cross entropy for the training and validation sets over the fifty epochs used for training. As shown in the figure, the metrics stabilize, and the best values for the validation set (lowest loss and highest accuracy) are observed at epoch 21. The weights corresponding to this epoch were saved and used for testing the model and calculating the following metrics.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Layer** & **Hyperparameter** & **Values Tested** \\ \hline Dense & Units & [8, 55, 150, 300, 500] \\ Dense & Activation Function & [’linear’, ’relu’, ’tanh’] \\ Dropout & Rate & [0.0, 0.15] \\ Optimizer & Learning Rate & [0.01, 0.001] \\ \hline \hline \end{tabular}
\end{table}
Table 6: Best hyperparameters found.  
Figure 12: Examples of correctly classified images.  
Figure 13: Accuracy and categorical cross-entropy over the epochs.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Layer** & **Hyperparameter** & **Values Tested** \\ \hline Dense & Units & [8, 55, 150, 300, 500] \\ Dense & Activation Function & [’linear’, ’relu’, ’tanh’] \\ Dropout & Rate & [0.0, 0.15] \\ Optimizer & Learning Rate & [0.01, 0.001] \\ \hline \hline \end{tabular}
\end{table}
Table 5: Hyperparameters tested in Keras tuner.  
After training the model, the model was tested with new images (the test set), and the confusion matrix in Table 7 was obtained. This matrix allows a deeper analysis of the types of errors made by the classifier and the number of incorrect images misclassified in each class.  
Table 8 shows the classification metrics obtained for the trained model. Overall, the model achieved an accuracy of 81.4%. However, since the dataset is imbalanced due to the much lower number of images without graffiti, the balanced accuracy, which corresponds to the average recall obtained in each class, has slightly decreased to 77.2%.  
The precision explains how many images predicted with a positive class were correctly classified. This metric indicates that the class 'illegal graffiti' is the one that is more times mispredicted, and the class with the lowest false positive rate is'street art'.  
The recall metric corresponds to the ratio between correct predicted positive observations and the total number of observations. This metric indicates that, from all the images without graffiti, only 56% were correctly classified. The pictures with illegal graffiti have the highest proportion of true positives.  
However, the metric that considers false positives and false negatives is the F1-score, the harmonic mean of precision and recall. We can see that the class with the best predictions overall is'street art', and the worst are the images without graffiti, most likely due to the smaller number of pictures used without graffiti in the model training.