# Classification and Causes Identification of Chinese Civil Aviation Incident Reports
## 4 Cause Extraction
### Experimental Results  
As shown in Figure 3, the dataset was divided into a training set and a test set, with each take part of 50%. The F1-score was adopted to evaluate the performance. We conducted cross-validation on the training set first. All reports were randomly divided into five sets, and each one built keywords and rules independently. The updating of keywords and rules is an iterative process. The keywords and rules in the first version were derived from a manual analysis process for each set of reports. The rules that led to error detection were modified based on their performance in other situations until an optimal result was reached. Then, five sets of rules, which performed best in the corresponding report sets, were used for detecting other sets, namely, for performing cross-validation. Table 9 reports the F1 scores for the 5-fold cross-validation. Although using keywords and rules from other reports, the minimum F1 score for each set was above 0.80, which is an acceptable result in terms of cause identification. This means that these rules can apply to other reports in the database. In addition, the system is sufficiently robust, and the best F1 scores were all above 0.90. After cross-validation, five sets rules were integrated and tuned to their best performance on the whole training set. The results of cause detection on the test set were the final evaluation of this system. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Part** & **Sentence Context in Chinese** & **English Version** & **Identified Causes** \\ \hline \hline \multirow{6}{*}{3} & \multirow{6}{*}{200} & Verified, the incident occurred at a distance of 95 km from the opposite & \multirow{6}{*}{-} \\  & & & aircraft and the distance between the two aircraft was in line with the safe & \\  & & & spacing and no conflict arose. & \\ \hline \hline \multirow{6}{*}{4} & \multirow{6}{*}{200} & Due to the thunderstorm, most of the crew’s attention was focused on & \multirow{6}{*}{-} \\  & & performing the thunderstorm detour & & \\  & & and was distracted from the task, & & \\  & & resulting in the incorrect setting of the & & \\  & & pre-selected altitude. & & \\ \hline \hline \multirow{6}{*}{5} & \multirow{6}{*}{200} & Secondly, the crew did not strictly & \multirow{6}{*}{H2, H4} \\  & & follow the SOP standard shouting and & & \\  & & operating procedures, and the control & & \\ \cline{1-1}  & & area handover did not confirm the & & \\ \cline{1-1}  & & altitude instructions and cross checking & & \\ \cline{1-1}  & & altitude settings. & & \\ \hline \hline \multirow{6}{*}{Analysis} & \multirow{6}{*}{6} & (1) There was a thunderstorm on the & \multirow{6}{*}{H5} \\  & & route and the crew was busy applying & \multirow{6}{*}{H5} \\  & & for a go-around, which distracted their & \\ \cline{1-1}  & & attention and led to the crew was & \\ \cline{1-1}  & & distracted by the “wrong” reverse & \multirow{6}{*}{H5} \\ \cline{1-1}  & & altitude and did not confirm the altitude & \\ \cline{1-1}  & & with the controller immediately. & \\ \hline \hline \end{tabular}
\end{table} 
Table 8: _Cont._Considering the prevalence of synonyms, we used the Word2 Vec technique to expand the keywords (mainly AKs). Table 10 summarizes the precision, recall, and F1 score of the test set. In general, even if the rules were derived from other reports, the results yielded F1 scores of 0.90 or higher. However, there is a clear difference between precision and recall. Higher recall (0.95) means that almost all causes can be detected, while the relatively low precision demonstrates an unremarkable accuracy of causes identification. Figure 3: The framework of the experiment for cause identification.