# An Initial Machine Learning-Based Victim's Scream Detection Analysis for Burning Sites
## 3 Methodology
### Performance Metrics  
We evaluated the performance of the classifiers discussed in the previous section based on the following metrics. The best model is one that provides higher overall accuracy rates, with minimal misclassifications, and that is stable, or convergent, through the training and validation processes. The k-fold technique was applied to the SVM model to optimize the hyperparameters to achieve highest rates in training and testing [57].  
* Accuracy: The accuracy metric measures how well the model performed when predicting samples from the dataset. This is computed using the following formula: \[\textit{Acc.}\left(\%\right)=\frac{\textit{Number of correct predictions on data samples}}{\textit{Total number of data samples}}\times 100\]
* Precision and recall: These factors give an insight of how well the model is performing when classifying individual classes. Precision quantifies the proportion of correctly positive identification and is calculated by \[\textit{Precision}=\frac{\textit{True Positive }\left(TP\right)}{\textit{True Positive }\left(TP\right)+\textit{False Positive }\left(FP\right)}\]Recall measures the proportion of actual positives identified correctly. Recall is calculated by \[Recall=\frac{\text{\it True\ Positive\ (TP)}}{\text{\it True\ Positive\ (TP)}+\text{\it False\ Negative\ (FN)}}\]
* F1 score: The F1 score can be interpreted as a weighted average of the precision and recall and is computed with \[\text{F1}=2\times\frac{\text{\it Precision\ }\times\text{\it Recall}}{\text{\it Precision }+\text{\it Recall}}\]
* Confusion matrix (_CM_): CM is a two-dimensional array of numbers where one axis represents the true labels of the classes, and the other axis represents the predicted labels by the classifier. This array or matrix shows the numbers of correctly predicted samples and the incorrect ones for each of the classes separately.
* Accuracy curve: This metric is popular among the deep neural network architectures. This represents the accuracy plot as a function of the number of iteration (epoch). This is a good measure to detect over-fitting and under-fitting issues. The under-fitting problem occurs when a model has a high bias, meaning that the features are unable to find a relationship with the target classes. Over-fitting problem occurs when the model performs extremely well on the training data but performs poorly on the unseen data (in this case validation/test data).
* Loss curve: Loss curve serves the same purpose as accuracy curve, where loss during the training and validation is plotted as a function of the number of iterations. The categorical cross-entropy loss is a loss function used in multiclass classification problems, which calculates the cross-entropy loss between the class labels and the predictions.
* Receiver operating characteristic (ROC) Curve: This is a highly useful tool when predicting the probability of a binary outcome. It plots the false positive rate (\(x\)-axis) versus the true positive rate (\(y\)-axis) for several candidate threshold values between 0.0 and 1.0. Since this problem is a multi-class problem and our focus is to classify screams, we used the One-versus-Rest (OvR) approach and considered'scream' as one class and the rest of the classes combined as another class. The area under the curve (AUC) can be used as a summary of the model effectiveness.
* _K_-fold cross-validation score: When the same samples are repeatedly fed to the networks, the model tends to overfit the data. _K_-fold cross-validation technique is used to avoid overfitting. It divides the whole training set into _k_-folds where _k-1_ folds are used for training, and the remaining fold is used for validation. The model is trained and validated a total of _k_-times, each using a different fold for the validation and training set. The final score is then calculated by averaging the resulting _k_-scores. We have used a \(k\) = 10 folds on the 70% dataset (combined training and validation set), where 9 folds were used for training and the remaining fold was used for validation.