## 4 Empirical Analysis
### Results of Empirical Analysis  
In the division of the machine learning model dataset, this paper adopts the 10-fold cross-validation, dividing the dataset into 10 parts, and taking turns to use 9 of them as the training set and 1 as the test set. The process is repeated 10 times, and we use the average of the results as the estimation of the algorithm accuracy. For one thing, it enhances the generalization ability of the model. For another thing, it avoids the overfitting situation. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Year** & **Number of ST Companies** & **Total Number of Listed Companies** & **Ratio** \\ \hline
2012 & 16 & 2494 & 0.0064 \\
2013 & 23 & 2489 & 0.0092 \\
2014 & 35 & 2613 & 0.0134 \\
2015 & 42 & 2827 & 0.0149 \\
2016 & 58 & 3052 & 0.0190 \\
2017 & 55 & 3485 & 0.0158 \\  
This paper performs ten times the 10-fold cross-validations and averages the results to reduce the chance and increase the confidence of the results. As for the evaluation indexes of the early warning effect of machine learning models, this paper selects the average accuracy, the prediction accuracy of non-ST listed companies, the prediction accuracy of ST listed companies and Area Under Curve (AUC) as the evaluation indexes. In the dichotomous classification, the prediction results will appear in the following four cases. 1. True Positive (TP): Positive samples predicted by the model as positive;
2. False Positive (FP): Negative samples predicted by the model as positive;
3. False Negative (FN): Positive samples predicted by the model as negative;
4. True Negative (TN): Negative samples predicted by the model as negative;  
According to the four prediction results, this paper calculates the evaluation indicators of the machine learning model. The average accuracy (ACC) is calculated by Equation (14), which represents the share of the number of correctly predicted samples to the total number of the samples. \[\text{ACC}= \frac{\text{TP}+\text{TN}}{\text{TP}+\text{FP}+\text{FN}+\text{TN}} \tag{14}\]  
The prediction accuracy rate of non-ST listed companies, also known as sensitivity (SEN), is calculated by Equation (15), which represents the percentage of samples with correct predictions among all samples that are truly non-ST listed companies. \[\text{SEN}= \frac{\text{TP}}{\text{TP}+\text{FN}} \tag{15}\]  
The prediction accuracy rate of ST listed companies, also known as specificity (SPE), is shown in Equation (16), which represents the percentage of samples with correct predictions among all samples that are truly ST listed companies. \[\text{SPE}= \frac{\text{TN}}{\text{TN}+\text{FP}} \tag{16}\]  
AUC is the area under the line of the Receiver Operating Characteristic (ROC) Curve, the area which is chosen to measure the accuracy of the dichotomous classification model. The larger the value of the AUC area is, the higher the classification accuracy of the model. When the value of the AUC area is less than 0.5, the model will almost lose its predictive effect. the AUC calculation formula is shown in Equation (17). \[\text{AUC}=\frac{\sum_{i\in\text{ positiveClass}}rank_{i}-\frac{m(1+m)}{2}}{m*n} \tag{17}\]  
\(rank_{i}\) represents the serial number of the \(i^{\text{th}}\) sample. \(m\) and \(n\) represent the number of positive samples and negative samples, respectively. \(\sum_{i\in\text{ positiveClass}}\) means the numbers of positive samples are added up. In this paper, AUC, as well as the value of SPE, will be prioritized when determining the effect of early warning. This is because AUC is used to measure the overall performance of a model in identifying the financial crisis of the sample companies. As the prediction accuracy of the sample of ST listed companies, SPE is more important than the prediction accuracy of the sample of non-ST listed companies, because the goal is the identification of the listed companies in financial crisis.