## 5 Experimental Results
### Evaluation of Classifiers per Dataset  
#### 5.3.1 Default Classifiers  
The reported accuracy of several classifiers is summarized in Table 3. The paper associated with each dataset is selected for this comparison and is to be used as a reference of the expected performance. Nonetheless, not all classifiers are evaluated for each dataset.  
Our measurements when evaluating multiple classifiers using the existing datasets are summarized in Table 4. Although only the top 10 features with the highest entropy for each dataset reported by the Weka audio analysis tool could be selected, all the features available in LibROSA have been used as input vectors for the classifiers. Whereas our measurements show that most of the classifiers improve the literature's results (Table 3), our SVM classifier presents a significantly lower accuracy than expected. This lack of accuracy can be associated with the configuration of the SVM classifier since we used the default configuration in our tests and its configuration is not usually reported.  
The relation between the number of categories per dataset and the classifier accuracy is exposed in Table 4. The dataset ESC-50 composed of 50 categories [2] results in the lowest classifiers' performance compared to the others. This low performance can also be seen in the number of features whose entropy is null, and, therefore, discarded. As a result, a dataset is recommended to have a limited number of categories. In fact, the number of categories should be as small as possible according to [1; 36].  
Table 4 also depicts the average execution time of each classifier to process the feature vectors from all frames. The average time computation and accuracy of each classifier is done using 10 iterations. There exists a significant difference between the computational time demanded by each classifier. Whereas ANN only demands few microseconds, k-NN presents the highest demand achieving around two milliseconds. Such disparity must be considered when selecting the classifiers to be evaluated on an embedded system taking into account the constrained nature of these devices.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Classifier**} & \multicolumn{2}{c}{**BDLib**} & \multicolumn{2}{c}{**ESC-10**} & \multicolumn{2}{c}{**ESC-50**} & \multicolumn{2}{c}{**UrbanSound**} \\ \cline{2-9}  & **Accuracy [\%]** & **Timing [ms]** & **Accuracy [\%]** & **Timing [ms]** & **Accuracy [\%]** & **Timing [ms]** & **Accuracy [\%]** & **Timing [ms]** \\ \hline k-NN & 59.58 (15.22) & 1.98 (0.69) & 73.50 (6.26) & 1.54 (0.44) & 46.90 (4.41) & 15.13 (0.59) & 42.83 (8.09) & 5.91 (0.17) \\ Naive Bayes & 62.08 (14.76) & 1.04 (0.06) & 69.50 (8.64) & 0.95 (0.09) & 45.00 (50.01) & 5.38 (0.69) & 53.50 (5.74) & 1.21 (0.15) \\ ANN & 52.92 (12.27) & 1.08 (0.02) & 67.50 (8.89) & 0.19 (0.08) & 21.20 (2.82) & 0.35 (0.06) & 39.17 (4.73) & 0.22 (0.01) \\ SVM & 30.00 (8.52) & 0.71 (0.19) & 40.00 (12.47) & 0.73 (0.25) & 11.80 (4.05) & 0.92 (0.11) & 29.17 (4.86) & 0.65 (0.07) \\ Decision Trees & 29.17 (10.21) & 0.55 (0.07) & 39.00 (8.09) & 0.54 (0.08) & 7.50 (2.37) & 0.67 (0.04) & 26.50 (4.74) & 0.64 (0.05) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Measured classifiers’ accuracy per dataset and average execution time of the classifiers with default configuration. Accuracy and timing values are expressed in percentage and milliseconds, respectively. The values in brackets are the Standard Deviation (SD).  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Classifier** & **BDLib**[1] & **ESC-10**[2] & **ESC-50**[2] & **UrbanSound**[3] \\ \hline k-NN & 45.5\% & 66.7\% & 32.2\% & - \\ Naive Bayes & 45.9\% & - & - & - \\ ANN & 54.0\% & - & - & - \\ SVM & 53.7\% & 67.5\% & 39.9\% & \(\approx\)70\% \\ Decision Trees & - & - & - & \(\approx\)48\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Classifiers’ accuracy per dataset reported in the literature.