# Towards Domain-Specific Knowledge Graph Construction for Flight Control Aided Maintenance
## 5 Experiments  
### Experimental Setup  
We conducted experiments to verify the effectiveness of the entity-relation extraction methods proposed. The experiments were implemented by PyTorch 0.4.1 and CUDA 9.1. The hardware platform was composed of two Nvidia Geforce RTX 3090 (24 G) GPUs and an Intel 10980 XE CPU. The testing input was \"AIRBUS Training and Flight Operations Support and Services\" which is an unstructured PDF and describes the composition of the flight control system. After de-noising, we obtained 1890 named entities and 4878 relations (4092 relations belonging to _Other_). Entity-relation extraction is based on sentences such that 90% of sentences were selected randomly for training, and the remaining 10% sentences were left for testing. Training epochs were set to 80, and the learning rate was \(10^{-3}\). We recorded \(P\) (Precision), \(R\) (Recall) and F1 score to evaluate the performance of NER and relation extraction. \(P\), \(R\), and F1 were calculated by Equations (10)-(12), respectively. \[P=\frac{TP}{TP+FP} \tag{10}\]  
\[R=\frac{TP}{TP+FN} \tag{11}\]  
\[F1=2\times\frac{P\times R}{P+R} \tag{12}\]  
\(TP\) (true positive), \(FP\) (false positive) and \(FN\) (false negative) in Equations (10) and (11) are, respectively, the number of entities (or relations) extracted correctly, false alarms, and missing entities (or relations). Table 1 provides important dimension of embeddings leveraged in both the entity recognition and relation extraction algorithms (referring to Equations (1)-(4), (8) and (9)). \begin{table}
\begin{tabular}{c c} \hline
**Embedding** & **Dimension** \\ \hline \(WORD_{i}\) & 50 \\ \hline \(POS_{i}\) & 20 \\ \hline \(PHRASE_{i}\) & 30 \\ \hline \(HF_{i}\) & 10 \\ \hline \(Head_{i}\) & 10 \\ \hline \(TailP_{i}\) & 10 \\ \hline \end{tabular}
\end{table} 
Table 1: The dimension of important embeddings. From Tables 2-5, the performance of named entity recognition improves in general. Adding information of part of speech and information of domain-specific features can help the model to better distinguish entities. We provide two sentences as an example to show the progress of entity recognition. 1. If the aircraft reaches the pitch attitude protection nose up limit, then the flight control computer will override pilot demand and keep the aircraft within the safe flight limit. 2. In case of failure of both elec 1 servo control, then elec 2 does the computation and control its servo control. In the first sentence, \"flight control computer\" is an entity that can be correctly recognized if integrating information from part of speech and domain-specific features. However, if only using embeddings from word2 vec, the model can only recognize \"computer\", making a mistake on entity boundary. In the second sentence, the last two words \"servo control\" is an entity. Again, it cannot be recognized as an entity if only using embeddings from word2 vec. Table 5 integrates results from both the machine-learning-based algorithm and the rule-based algorithm. As the _Power_ class is special, the rule-based method already covers all possible cases such that all entities of _Power_ can be recognized accurately. Combining all results (the last row of Table 5), we think that the performance of named entity recognition reaches a reasonable level. In addition, we have noticed the weakness of the current method, especially when recognizing entities of the _Function_ class. Testing results on all the three criteria are not good compared with results on other entity classes. After a careful study, we find that entities belonging to _Function_ are diverse from each other. Training samples are not abundant enough to extract a variety of diverse features. Hence, missing recognition and wrong entity boundaries are more prone to happen compared with recognizing entities of other classes. For example, we find that the model always recognizes \"rudder deflection\" rather than \"rudder deflection limitation\" due to both of them being a kind of function.