## 4 Experiments
### Classifying Vowel Productions Based on Self-Reported VFI-1 Scores  
As indicated above, the goal of this experiment was to classify sEMG signals during vowel productions from subjects with self-reported vocal fatigue as a representation of real-life vocal fatigue. This experiment included a much larger data set of 88 subjects that included data from study 1 and study 2. Study 2 did not require subjects to simulate pressed phonation nor produce non-speech gestures. Furthermore, the challenge was to perform this classification under the inter-subject constraint. That is, for the previous cases, the tests were performed under the more typical sample-based cross-validation, where random samples from the entire pool are set apart for testing. Instead, for this experiment, we used a leave-one-out (LOO) approach, where if \(N\) was the number of vocally healthy subjects and \(M\) the number of vocally fatigued subjects, a total of \(M+N-1\) subjects were successively selected for training and validation, while the subject _left-out_ was used for testing. The only question remaining was that of how to separate the two groups of \(N\) vocally healthy and \(M\) vocally fatigued subjects. According to [5], the mean (standard deviation) of VFI-1 score for patients with dysphonia was 24.47 (9.76) and for vocally healthy individuals, this was 5.16 (4.58). Thus, one approach would be to derive two boundaries, VFI-1 \(\geqslant\) 15 (or \(\approx\) 24.47 - 9.76) for vocal fatigue and VFI-1 \(\leqslant\) 10 (or \(\approx\) 5.16 + 4.58) for normal voice. However, these two boundaries would have left subjects with scores between 10 and 15 in a \"gray area\" of VFI-1 scores investigated. Moreover, for the early detection of vocal fatigue, it is very important that the subjects in \"gray area\" be taken into consideration. Thus, we initially arbitrarily decided to use a VFI-1 score of 10 as the single threshold separating the two groups of test subjects. This threshold of 10 resulted in 26 vocally fatigued (23 teachers) and 62 vocally healthy subjects (14 teachers) in the dataset. In terms of the total number of samples, there were 4174 vowel productions from those with known vocal fatigue (positive samples) and 9979 normal vowel productions (negative samples). The results of the LOO classification are shown in Table 3.  
Table 3 shows the validation results which achieved an accuracy of 93.48%. This high accuracy in validation indicated that there was a clear distinction between the samples and the classifier was very successful in discriminating the two classes. However, after conducting a testing experiment--using the LOO approach--the same accuracy dropped to 58.13%, with most of the error coming from false positives (FPs). This lower-than-expected accuracy for testing may be related to a difficulty for the classifier to generalize. In that sense, it is necessary to highlight that our data were not evenly balanced. In fact, there were \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Actual Positive** & **Actual Negative** \\ \hline
**Predicted Positive** & 90.60\% & 9.80\% \\
**Predicted Negative** & 9.40\% & 90.20\% \\
**Validation Accuracy** & 90.40\% & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Confusion matrix for Positive vs. Negative detection of simulated fatigue (pressed vowels) among 57 test subjects under the _intra-subject_ constraint. more than twice as many negative than positive samples. Typically, this imbalance causes a bias in the system towards the negative class, which can be easily observed by a much lower sensitivity (true positive rate) with respect to specificity (true negative rate). In this experiment, sensitivity was 0.23 and specificity was 0.73. In essence, while most normal vowel productions were correctly classified, about one fifth of the vowel productions from speakers with self-reported vocal fatigue were misclassified. Three potential limitations could explain this lack of generalization: first, as already mentioned, our data were imbalanced.