# Towards Domain-Specific Knowledge Graph Construction for Flight Control Aided Maintenance
## 5 Experiments
### Experimental Setup
#### 5.2.2 Relation Extraction  
The results of relation extraction are given, respectively, in Tables 6-9. The criteria demonstrated are still \(P\) (Precision), \(R\) (Recall), and F1 score. Let us combine the results to explain one by one. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Types of Relation** & **Precision** & **Recall** & **F1** \\ \hline _defined relation_ & 0.89 & 0.81 & 0.85 \\ \hline _Other_ & 0.98 & 0.99 & 0.98 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Binary relation extraction. Remember that we leverage a two-stage relation extraction procedure, where the first stage is a binary classification problem. We involve this stage aiming at eliminating the impact generated by excessive negative samples. Results in Table 6 show the model can well differentiate (up to 98%) the cases having no defined relation, and meanwhile it can correctly keep more than 80% defined relations. During the second stage, we train a model to distinguish different types of defined relations. Table 7 provides the results of the two-stage method. It can be seen that with prior knowledge on correct pair of entities, the model can well recognize the type of relations: the average Precision reaches 0.86 and the average F1 reaches 0.81. We also noticed that relation _Achieve_ cannot be extracted by the model. The problem arises from the binary classification. At this stage model extract common features of all existence relations. Unfortunately, samples of _Achieve_ are diverse and the number is limited such that the model is prone to make a mistake. To demonstrate the effectiveness of the two-stage method, we also give out the results by directly training the model (named \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Types of Relation** & **Precision** & **Recall** & **F1** \\ \hline _ConsistOf_ & 1.00 & 0.75 & 0.86 \\ \hline _Control_ & 0.80 & 1.00 & 0.92 \\ \hline _Achieve_ & 0.00 & 0.00 & 0.00 \\ \hline _Acronym_ & 0.71 & 1.00 & 0.83 \\ \hline _Drive_ & 0.56 & 1.00 & 0.92 \\ \hline _SendMsgTo_ & 1.00 & 0.50 & 0.67 \\ \hline _ConnectIo_ & 1.00 & 1.00 & 1.00 \\ \hline _LocatedOn_ & 1.00 & 1.00 & 1.00 \\ \hline _Average_ & 0.86 & 0.77 & 0.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Two-stage method: relation extraction results based on annotated entities. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Types of Relation** & **Precision** & **Recall** & **F1** \\ \hline _ConsistOf_ & 1.00 & 0.25 & 0.40 \\ \hline _Control_ & 1.00 & 1.00 & 1.00 \\ \hline _Achieve_ & 0.00 & 0.00 & 0.00 \\ \hline _Acronym_ & 0.71 & 1.00 & 0.83 \\ \hline _Drive_ & 0.80 & 0.67 & 0.73 \\ \hline _SendMsgTo_ & 0.75 & 0.75 & 0.75 \\ \hline _ConnectIo_ & 0.67 & 1.00 & 0.80 \\ \hline _LocatedOn_ & 0.67 & 1.00 & 0.80 \\ \hline _Average_ & 0.75 & 0.68 & 0.71 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: One-stage method: relation extraction results based on annotated entities. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Types of Relation** & **Precision** & **Recall** & **F1** \\ \hline _ConsistOf_ & 0.75 & 0.75 & 0.75 \\ \hline _Control_ & 0.67 & 0.50 & 0.57 \\ \hline _Achieve_ & 0.00 & 0.00 & 0.00 \\ \hline _Acronym_ & 0.43 & 0.60 & 0.50 \\ \hline _Drive_ & 0.75 & 0.50 & 0.60 \\ \hline _SendMsgTo_ & 1.00 & 0.25 & 0.40 \\ \hline _ConnectIo_ & 1.00 & 0.50 & 0.67 \\ \hline _LocatedOn_ & 0.50 & 0.50 & 0.50 \\ \hline _Average_ & 0.61 & 0.45 & 0.52 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Relation extraction based on recognized entities. one-stage) for comparison. The results are provided in Table 8. It can be seen that by the one-stage method, the average performance decreases by more 10 percentages. Note that results in Table 7 are obtained based on correct entities. To obtain the final results for knowledge graph construction, it is necessary to combine with the named entity recognition. Table 9 provides the final results on relation extraction. Compared with results in Table 7, there exists an apparent performance decrease. Faults in NER directly encumber the relation extraction.