## 4 Results and Discussion
### Performance of the Proposed System
It is worth noting that, overtraining and overfitting issues were also studied to assess the effectiveness of the developed models. Specifically, in order to control the aforementioned phenomenon, the \(k\)-fold cross validation (\(k\) = 7) technique was adopted and train accuracies were also compared with those achieved in the test phase. Note that the following data separation was employed: 70% for train and 30% for test. As can be seen from Figure 11, all the proposed models were not deficient of overtraining or overfitting and provided good generalization abilities, reporting a small standard deviation and a maximum gap between train and test average accuracy of only 1%. Furthermore, networks were trained until the converge of the cross-entropy function was observed. As an example, Figure 12, reports the training phase of the best classifier proposed in this study, i.e., MLP\({}_{1}\). However, it is also worth mentioning that the topology of the classifiers, learning and training parameters were set-up performing several experimental tests according to a _trial and error_ approach. Hence, experimental results showed that the proposed MLP\({}_{1}\) classifier achieved the best detection performance when compared with other machine learning techniques (i.e., MLP\({}_{2}\), MLP\({}_{3}\), AE\({}_{1}\), AE\({}_{2}\), AE\({}_{3}\), 1D-CNN, SVM) as well as other previous shadow detection approaches [9; 11], reporting accuracy and AUC rates up to 84.63 \(\pm\) 0.63% and 89 \(\pm\) 0.8%, respectively. It is interesting to note that MLP outperforms AE and 1D-CNN classifiers (that belongs to more advanced machine learning architectures [65]). This is possible due to the limited size of input space (only 36 features). Indeed, DL-based approaches are typically used to big data with a large input dimension. Here, the proposed AE may cause an over compression of features and hence a loss of significant information; while the hierarchical learning representation of a standard CNN is too complex to cope with the pixel-based classification task of this study. For these reasons, common machine learning algorithms with a simpler architecture (i.e., MLP) achieved better results. It is worth noting that also the developed AE and 1D-CNN achieved very good results. In particular, AE\({}_{3}\) reported detection accuracy of 77.91 \(\pm\) 0.72%, whereas 1D-CNN reported accuracy of 75.8 \(\pm\) 1.2%. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F-Measure** & **Accuracy** \\ \hline MLP\({}_{1}\) & 82.72 \(\pm\) 0.77\% & 87.52 \(\pm\) 0.4\% & 85.05 \(\pm\) 0.57\% & 84.63 \(\pm\) 0.63\% \\ MLP\({}_{2}\) & 79.47 \(\pm\) 0.97\% & 85.55 \(\pm\) 0.66\% & 82.39 \(\pm\) 0.69\% & 81.71 \(\pm\) 0.77\% \\ MLP\({}_{3}\) & 82.6 \(\pm\) 0.66\% & 86.6 \(\pm\) 0.45\% & 84.55 \(\pm\) 0.53\% & 84.19 \(\pm\) 0.56\% \\ AE\({}_{1}\) & 74.27 \(\pm\) 0.99\% & 81.14 \(\pm\) 1.47\% & 77.55 \(\pm\) 1.03\% & 76.51 \(\pm\) 1.04\% \\ AE\({}_{2}\) & 75.5 \(\pm\) 1.99\% & 81.44 \(\pm\) 1.42\% & 78.36 \(\pm\) 1.71\% & 77.51 \(\pm\) 1.89\% \\ AE\({}_{3}\) & 75.67 \(\pm\) 0.77\% & 82.29 \(\pm\) 0.75\% & 78.84 \(\pm\) 0.66\% & 77.91 \(\pm\) 0.72\% \\
1D-CNN & 73.5 \(\pm\) 1.91\% & 81.0 \(\pm\) 2.3\% & 76.90 \(\pm\) 1\% & 75.80 \(\pm\) 1.2\% \\ SVM & 62.5 \(\pm\) 3.88\% & 58.6 \(\pm\) 9.64\% & 59.93 \(\pm\) 3.85\% & 61.27 \(\pm\) 2.18\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparative results of the proposed MLP, AE, 1D-CNN, SVM classifiers evaluated on the test sets. All the outcomes are reported as mean value \(\pm\) standard deviation. Figure 11: Accuracy values achieved on the train and test sets, for each developed classifier. The black dot represents the average accuracy, while the vertical red line denotes the standard deviation. Figure 10: ROC curves of the proposed MLP, AE, 1D-CNN, SVM classifiers. Note that the MLP\({}_{1}\) achieved the highest AUC = 0.92 \(\pm\) 0.005.