# A Named Entity and Relationship Extraction Method from Trouble-Shooting Documents in Korean
## 7 Experiment and Results  
This paper analyzed equipment maintenance documents used in domestic plant enterprises. In total, 10 out of 17 equipment documents were selected, excluding duplications. Troubleshooting data were divided into \"failure mode\" and \"repair action\", and each data set was organized into sentence units. The data on failure mode consisted of 451 sentences, and the data on repair actions consisted of 449 sentences  
In the experiment, combinations of two morpheme analyzers and two NER approaches were compared to find the best match for the troubleshooting sentences in Korean.  
Kkma and Mecab as the morpheme analyzer generated different numbers of tokens. For the failure mode sentences, Kkma generated 1956 tokens, while Mecab generated 1774 tokens. For the repair action sentences, Kkma generated 4551 tokens, while Mecab generated 4168 tokens.  
When examining the sentences constituting each document, some used a similar vocabulary, but some used many different expressions, making the sentences inconsistent. In addition, several terms were sometimes used in different sentences to represent the same equipment part. In order to determine whether a NER approach can resolve the semantic heterogeneity of terms, the KLUE NER and CRF approaches were trained and tested.  
Domain experts provided correct BIO tags for the tokenized sentences. For the failure mode, three types of tags, except for the \"ETC\" tag, were used: \"PART\", \"FAILURE\", and \"O\". For the repair action, four types were used: \"PART\", \"FAILURE\", \"ACTION\", and \"O\". In total, 70% of the total sentences were randomly selected to train the KLUE NER model and CRF model. The other 30% of the total sentences were used to test the trained models.  
The performance of the combination of the morpheme analyzer and the NER approach was measured by the precision, recall, and F1 score, as defined in the following equations.  
\[\textit{precision}=\frac{\textit{True Positive}}{\textit{True Positive}+\textit{False Positive}} \tag{2}\]  
\[\textit{recall}=\frac{\textit{True Positive}}{\textit{True Positive}+\textit{False Negative}} \tag{3}\]  
Figure 4: Screenshots of the proposed rule-based system.  
\[F1\ score=\frac{2\times precision\times recall}{(precision+recall)} \tag{4}\]  
_True positive_ denotes the number of correct answers of the trained model for the entities, except \"O\". _False positive_ means the number of wrong answers of the trained model for the entities, except \"O\". _False negative_ means the number of wrong answers of the trained model for the \"O\"-tagged terms.  
Table 9 shows the experimental results. The final F1 score of KLUE-NER with Mecab was 0.8235 for the failure mode data and 0.6888 for the repair action data. The relatively low results for the repair action data may be attributed to the fact that, despite data pre-processing, some unnecessary expressions were still included in the repair action data.  
For the recognized entities, relation sets were established by applying the rules that considered the dependency parsing results. Correct relation sets were given by domain experts. The performance results of the knowledge extraction using the proposed rules in Sections 6.1 and 6.2 are presented in Table 10. The number of sentences with accurately recognized entities and relations was 133 out of a total of 179 sentences (74.3%). The proposed method was successful in 56 out of 65 failure mode-related sentences and 77 out of 114 repair action-related sentences, and it obtained a 86.15% and 67.54% accuracy, respectively.