## 3 Experiment
### Experimental Results and Analysis  
In the construction part of the multi-classification system, the classification problem is extended to four exact types of pathological voice: vocal cord nodules, vocal cord polyps, vocal cord edema, and vocal cord paralysis. The model evaluation measure is obtained by recall, specificity, \(G\) value, and \(F1\) value through 10-fold cross-validation. Meanwhile, each classifier training time is also presented to evaluate the data complexity. Table 6 shows the results of ten machine learning models before using FC-SMOTE to balance the dataset. The predictive ability of all models is inferior. Due to the class-imbalanced distribution of the sample in the MEEI database, most of the model evaluation measures are lower than 0.5, which may also be due to the current lack of work on the multi-classification of pathological voices. The model evaluation measures of each classifier after imbalanced handing using the FC-SMOTE algorithm are shown in Table 7. Additionally, the results of external validation are shown by a multi-class confusion matrix in Figure 5. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Actual Classes**} & \multicolumn{2}{c}{**Prediction Results**} \\ \cline{2-3}  & **Positive Class** & **Negative Class** \\ \hline Positive class & \(TP\) & \(FN\) \\ Negative class & \(FP\) & \(TN\) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The confusion matrix. \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & **LR** & **NB** & **DT** & **SVM** & **KNN** & **RF** & **XGBoost** & **GBDT** & **MLP** & **CNN** \\ \hline Recall & 0.31 & 0.41 & 0.25 & 0.28 & 0.33 & 0.30 & 0.27 & 0.33 & 0.30 & 0.33 \\ Specificity & 0.78 & 0.82 & 0.77 & 0.76 & 0.79 & 0.77 & 0.78 & 0.79 & 0.78 & 0.78 \\ \(G\) value & 0.48 & 0.58 & 0.44 & 0.46 & 0.51 & 0.46 & 0.45 & 0.50 & 0.48 & 0.47 \\ \(F1\) value & 0.41 & 0.41 & 0.29 & 0.48 & 0.42 & 0.44 & 0.40 & 0.34 & 0.42 & 0.41 \\ Time(s) & 0.45 & 0.23 & 0.44 & 0.32 & 0.27 & 7.81 & 6.25 & 11.29 & 19.25 & 25.49 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Evaluation measures of each machine learning model in the multi-imbalanced MEEI dataset. Figure 5: The multi-class confusion matrices of logistic regression(LR), naive Bayes (NB), decision tree (DT), support vector machine (SVM), K nearest neighbours (KNN), multi-layer perceptron (MLP) and random forest (RF) after FC-SMOTE processing. \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & **LR** & **NB** & **DT** & **SVM** & **KNN** & **RF** & **XGBoost** & **GBDT** & **MLP** & **CNN** \\ \hline Recall & 0.72 & 0.70 & 0.78 & 0.82 & 0.72 & 0.83 & 0.82 & 0.74 & 0.72 & 0.73 \\ Specificity & 0.92 & 0.90 & 0.93 & 0.95 & 0.92 & 0.95 & 0.95 & 0.93 & 0.92 & 0.90 \\ \(G\) value & 0.81 & 0.79 & 0.85 & 0.88 & 0.80 & 0.88 & 0.88 & 0.83 & 0.82 & 0.81 \\ \(F1\) value & 0.76 & 0.68 & 0.79 & 0.85 & 0.76 & 0.86 & 0.85 & 0.80 & 0.77 & 0.77 \\ Time(s) & 0.73 & 0.12 & 0.38 & 0.39 & 1.29 & 18.72 & 19.29 & 16.16 & 27.77 & 54.60 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Evaluation measures of each machine learning model in the multi-class balanced MEEI dataset. It can be seen from the comprehensive model evaluation measures of the ten classifiers in Table 7 that the VPD system with FC-SMOTE performs better in terms of the recall, specificity, \(G\) value, and \(F1\) value. Compared with the single model and the neural network model, the three ensemble models (RF, XGboost, GBDT) have the most significant improvement and have the best overall performance in model evaluation measures after the FC-SMOTE algorithm handing. The optimal classifier RF increases the recall to 0.83, the specificity to 0.95, the \(G\) value to 0.88, and the \(F1\) value to 0.86. As an ensemble learning model, RF performs better than single classifiers in pathological voice classification, which is also reflected in the latest review paper [6]. Meanwhile, the same effect is shown in two other typical ensemble learning models (GBDT, XGBoost).