# _Applied_
## 6 Evaluation of Classical Machine Learning for ESR: The MosAlc Approach
### The One-vs-All Approach  
One strategy to improve accuracy could be the reduction of the classes to be recognized. The minimal number of classes can be reduced to only two classes, which can be used to determine if an audio belongs to one associated class or not. This approach, called one-vs-all [42], is evaluated here. To set this approach, the features are extracted and labeled in a binary way: either \"the-associated-class\" or \"not-the-associated-class\". This is done separately for each class. The classifiers are then trained in a binary way. For instance, the classifiers are trained to distinguish the \"the-associated-class\" _dog bark_ from the UrbanSound dataset and audio labelled as the \"not-the-associated-class\" _no dog bark_. The number of audio samples labelled \"not-the-associated-class\" will be always higher than \"the-associated-class\". In the case of the ESC-10 and BDLib datasets, this number is exactly nine times more. This creates imbalance in the training data, causing the classifier to actually learn to recognize \"not the class\" instead of \"this class\". Note that, due to its original imbalance, the UrbanSound dataset is the most sensible one to suffer this effect. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Classifier** & **Classification Time [ms]** & \(\mathbf{F_{1}}\) **Micro** & \(\mathbf{F_{1}}\) **Macro** & \(\mathbf{F_{1}}\) **Weighted** \\ \hline SVM & 0.001 \(\pm\) 0.000 & 0.844 & 0.841 & 0.843 \\ Naive Bayes & 0.016 \(\pm\) 0.000 & 0.516 & 0.512 & 0.514 \\
**k-NN** & **0.597 \(\pm\) 0.023** & **0.876** & **0.872** & **0.873** \\ ANN & 0.002 \(\pm\) 0.001 & 0.821 & 0.820 & 0.821 \\ Decision Trees & 0.003 \(\pm\) 0.001 & 0.451 & 0.385 & 0.389 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: \(F_{1}\) scores and classification time using the ESC-10 dataset with a classical approach and 10 features. The solution with the highest \(F_{1}\) score is highlighted. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Classifier** & **BDLib [20]** & **ESC-10 [23]** & **UrbanSound [22]** \\ \hline k-NN & 45.5\% & 66.7\% & - \\ Naive Bayes & 45.9\% & - & - \\ ANN & 54.0\% & - & - \\ SVM & 53.7\% & 67.5\% & \(\approx\)70\% \\ Decision Trees & - & - & \(\approx\)48\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Classifiers’ accuracy per dataset reported in the literature (Source: [7]). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Classifier** & **Classification Time [ms]** & \(\mathbf{F_{1}}\) **Micro** & \(\mathbf{F_{1}}\) **Macro** & \(\mathbf{F_{1}}\) **Weighted** \\ \hline SVM & 0.001 \(\pm\) 0.000 & 0.624 & 0.609 & 0.622 \\ Naive Bayes & 0.017 \(\pm\) 0.000 & 0.250 & 0.251 & 0.252 \\
**k-NN** & **2.135 \(\pm\) 0.132** & **0.932** & **0.935** & **0.932** \\ ANN & 0.002 \(\pm\) 0.000 & 0.656 & 0.631 & 0.654 \\ Decision Trees & 0.001 \(\pm\) 0.000 & 0.353 & 0.214 & 0.291 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: \(F_{1}\) scores and classification time using the UrbanSound dataset with a classical approach and 10 features. The solution with the highest \(F_{1}\) score is highlighted. Figure 7 depicts the experimental results of \(F_{1}\) score using the one-vs-all approach with the different ML techniques using the UrbanSound dataset. This dataset is selected due to the relatively low accuracy obtained in the previous experiments. As depicted in Figure 7, the accuracy of the classifiers using the one-vs-all approach increases in comparison to the classical approach. As a result, the one-vs-all approach offers a major improvement in accuracy, with the \(F_{1}\) scores almost doubling for every ML technique. Due to the fact that the classifiers have to use the same set of features as in the traditional approach, their classification time is not altered. The exception is Naive Bayes, for which the classification time is reduced to 0.011 ms, half of the classification time measured in the previous section.