## 5 Experiment and Analysis
### Evaluation Methodology and Performance Measure  
In general, in the actual task, we need to evaluate the generalization error of the model, and then choose the model with the smallest generalization error. Therefore, it is necessary to use the testing set to test the discriminant ability of the model, and then take the test error of the testing set as an approximation of the generalization error. The testing set and training set are usually mutually exclusive, (i.e., the test samples do not appear in the training set and are not used in the training process). Therefore, this paper uses the hold-out method [40] to evaluate the model. The hold-out method directly divides the data set \(D\) into two mutually exclusive sets, namely the training set \(A\) and the testing set \(B\) (\(D=A\cup B\), \(A\cap B=\O\)) [41]. After the model is trained using training set \(A\), testing set \(B\) is used to evaluate the test error as an estimate of the generalization error. After evaluating the generalization performance of the model, it is necessary to measure the performance of the model with evaluation metrics. In this paper, four evaluation metrics are calculated, namely accuracy, precision, recall and f1-score. Their formulas can be seen in Equations (9)-(12) [22]:  
\[accuracy=\frac{TP+TN}{TP+FP+FN+TN}, \tag{9}\]  
\[precision=\frac{TP}{TP+FP}, \tag{10}\]  
\[recall=\frac{TP}{TP+FN}, \tag{11}\]  
\[f1-score==2\frac{precision\times recall}{precision+recall}, \tag{12}\]  
where \(TP\) means true positive, \(FP\) means false positive, \(TN\) represents true negative, and \(FN\) represents false negative. All of them are classified according to the combination of the real category and model prediction category [42]. Taking the binary classification as an example, the confusion matrix of the classification results is shown in Table 4. It is clear that the total number of samples equals the result of the formula \(TP+FP+TN+FN\). Different metrics directly reflect the impact of health monitoring tasks. For example, accuracy can directly reflect the number of correct and erroneous prediction results for all of the test samples. Precision can reflect a certain category of test samples, how many predictions are correct, and how many predictions are incorrect. For example, if in a testing set containing 100 samples of twisted rope, 90 are predicted to be twisted rope faults and 10 are classified as other faults, the precision for the twisted rope fault is 90%. Recall and precision are a pair of contradictory measurements. Recall shows how many predictions are correct in a certain class of prediction results. For example, if 100 prediction results are twisted rope faults, of which 90 test samples are actually twisted rope faults and 10 test samples are other faults, then the recall of the twisted rope fault is 90%. A good classifier maximizes \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Real Situation**} & \multicolumn{2}{c}{**Predicted Results**} \\ \cline{2-3}  & **Positive** & **Negative** \\ \hline Positive & \(TP\) & \(FN\) \\ \hline Negative & \(FP\) & \(TN\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The confusion matrix of the binary classification results. both precision and recall to make fewer incorrect prediction results, which is expressed in the f1-score. The f1-score is the harmonic average of precision and recall. The tail rope health monitoring in this paper is a multi-classification task. According to Section 4, different kinds of features, including normal (a), disproportional spacing (b), twisted rope (c), broken strand (d), and broken rope (e), should not normally be predicted incorrectly because their features are quite different. It may be difficult for classifiers to classify similar categories, for example, classifying between subcategories of broken strand (d): broken strand of the left rope (d1), broken strand of the right rope (d2), and broken strand of double ropes (d3).