# Towards Domain-Specific Knowledge Graph Construction for Flight Control Aided Maintenance
## 4 Knowledge Graph Construction for Flight Control System
### Named Entity Recognition and Relation Extraction
For each sentence, Transformer takes its embedding as the input, which is a tensor consisting of the embedding of every token. To combine the character of the flight control system, we customize word embedding delicately by adopting information from POS, domain-specific phrase, and whether it is a high-frequency word. Specifically, there are five steps:  
Step (1): The model word2 vec [44] is used to achieve an embedding for every token \(w_{i}\in S\). This process is represented by Equation (1), where \(WORD_{i}\) is the embedding obtained. \[WORD_{i}=embedding(w_{i}) \tag{1}\]  
Step (2): As entities are all noun phrases, we take POS into the embedding. For a sentence \(S=\{w_{1},w_{2},\dots,w_{n}\}\), we use the toolkit NLTK to obtain a corresponding sequence of POS: \(S_{pos}=\{pos_{1},pos_{2},\dots,pos_{n}\}\). Embedding related to POS is represented by Equation (2), where \(POS_{i}\) is the output embedding. \[POS_{i}=embedding_{pos}(pos_{i}) \tag{2}\]  
Step (3): We consider domain-specific phrases and add such information into embedding. Let \(S_{ph}=\{ph_{1},ph_{2},\dots,ph_{m}\}\) include the domain-specific phrases in the dictionary of terms. For a token \(w_{i}\in S\), if \(w_{i}\) matches one domain-specific phrase \(ph_{j}\), then \(PHRASE_{j}\) obtained from Equation (3) is the embedding of the domain-specific phrase. If \(w_{i}\) is not a domain-specific phrase, we leverage a default embedding. \[PHRASE_{i}=embedding_{ph}(ph_{i}) \tag{3}\]  
Step (4): We leverage the information of whether \(w_{i}\) is a high-frequency word. If \(w_{i}\) appears with high frequency, we denote it as \"True\". Otherwise, we denote it as \"False\". For a sentence \(S\), we can obtain a corresponding sequence \(\{hf_{1},hf_{2},\dots,hf_{n}\}\), where each \(hf_{i}\in\{True,False\}\). For each token \(w_{i}\), \(HF_{i}\) achieved from Equation (4) is the embedding referring to the information of high frequency  
\[HF_{i}=embedding_{bf}(hf_{i}) \tag{4}\]  
Step (5): Finally, all the partial embedding vectors are concatenated together as a complete one for the word \(w_{i}\):  
\[total\_embedding_{i}=[WORD_{i};POS_{i};PHRASE_{i};HF_{i}] \tag{5}\]  
Figure 6: Named entity recognition. For a sentence \(S\), embedding of each token is organized as a tensor which is the input of the Transformer encoder. The linear and CRF layer is used for label predication. The linear operation is for dimension transformation whose output is a sequence of vectors \(\{\vec{v}_{1},\vec{v}_{2},\ldots,\vec{v}_{n}\}\) corresponding to the \(n\) tokens of sentence \(S\). CRF takes charges of scoring each candidate label sequence. For example, consider \(\{\ell_{0},\ell_{1},\ell_{2},\ldots,\ell_{n},\ell_{n+1}\}\) is a candidate label sequence, where \(\ell_{0}\) and \(\ell_{n+1}\) are two auxiliary labels representing, respectively, the starting and ending tags of the sentence. The score of a candidate label sequence is obtained by Equation (6), where \(P_{i,\ell_{i}}\) is the probability that \(w_{i}\)'s label is \(\ell_{i}\), and \(T\) is a probability transition matrix with dimension \((n+2)\times(n+2)\). \[Score(\ell_{1},\ell_{2},\ldots,\ell_{n})=\sum_{i=1}^{n}P_{i,\ell_{i}}+\sum_{i= 0}^{n}T_{\ell_{i},\ell_{i+1}} \tag{6}\]  
Finally, a Softmax of all possible label sequences yields the probability (Equation (7)). The candidate label sequence with the highest probability is selected as the output. Figure 4:0  
Relation extraction is the task to predict the relation between two entities. As the importance of global semantic information, we consider Bi-LSTM + self-attention mechanism [32] to perform relation extraction. The attention mechanism adopted here aims at effectively obtaining key information. The schematic diagram of relation extraction is depicted in Figure 4:1  
The input \(\{e_{1},e_{2},S\}\) is a sentence \(S\) together with two entities \(e_{1}\) and \(e_{2}\).