# Natural Language Processing in Knowledge-Based Support for Operator Assistance
## 4 Result and Discussion  
To assess the performance of pre-trained taggers libraries, such as NLTK, SPACY, and Stanford, in the assembly domain corpus, and to determine if the newly created benchmark can effectively perform NLP tasks, the study utilized a ground truth consisting of 2752 tokens divided into four categories. The libraries' taggers were applied to the ground truth data to compare the tags produced by the libraries with the manually annotated ones. The tagging accuracy of the NLTK tagger compared to the ground truth was 90%, an improvement from the previous research's 87% accuracy with 100 initial sentences. In addition, recall 90%, precision 91%, and F1-score 90% were estimated for NLTK. The tagging accuracy of the SpaCy tagger on the ground truth was 93%, which is 3% more accurate than the NLTK tagger. This is consistent with the previous research, which achieved a 93% tagging accuracy with 100 initial sentences. In addition, recall, precision, and the F1-score were estimated as 93% for SpaCy. The Stanford tagger, as the newly added library, had 91% accuracy compared to the ground truth, 91% recall, 92% precision, and 91% F1-score. In the task of token recognition and comparison with the ground truth, the NLTK tagger achieved a perfect match by accurately identifying 2752 tokens, exhibiting a token accuracy rate of 100%. The NLTK tagger demonstrated the ability to correctly handle hyphenated compound words, such as \"oil-lubricated\", treating them as single tokens. In contrast, the SpaCy tagger successfully identified all tokens in the corpus but segmented hyphenated compound words into separate tokens, resulting in 2792 identified tokens. This discrepancy occurred because the SpaCy tagger treated hyphenated compound words as three distinct tokens, for instance, \"oil-lubricated\" being recognized as \"oil\", \"-\" and \"lubricated\". The assembly corpus contained a total of 36 hyphenated compound tokens. A similar issue arose for the Stanford library as the tagger identified 2786 tokens, recognizing 6 out of 36 hyphenated compound words. The accuracy of individual tags obtained by the considered libraries is presented in Figure 4. With regard to analyzing the three key parts of speech, namely _nouns_, _verbs_, and _adjectives_, SPACY achieved a high accuracy of 95% for _verbs_, 92% for _nouns_, and 78% for _adjectives_. NLTK achieved an accuracy of 88% for _verbs_, 91% for _nouns_, and 82% for _adjectives_. The considered accuracy for similar tags in the Stanford library was 94% for _verbs_, 94% for _nouns_, and 78% for _adjectives_. The three libraries demonstrated satisfactory accuracy in recognizing most tags. However, the NLTK and Stanford libraries exhibited the lowest accuracy of 62% specifically for _conjunction_ tags when compared to the other tags. Figure 4: Accuracy for each individual tag on the considered assembly corpus of NLTK, SPACY, and Stanford libraries taggers. The absolute numbers of mis-annotations for each type of token in NLTK, SPACY, and Stanford are presented in Figure 5. In NLTK, as shown in Table 4, the highest percentage of mis-annotations, at 38%, occurred for _conjunctions_. Specifically, 35 _conjunctions_ were mistakenly annotated as _prepositions_, and 24 were annotated as _adverbs_. Among the 251 _adjective_ tokens, 18% were mis-annotated, with 30 of them incorrectly labeled as _nouns_. Additionally, 11 out of the 12 mis-annotated tokens pertaining to _pronouns_ were erroneously annotated as _determiners_. There were instances of mis-annotations where _nouns_ and _verbs_ were mistakenly labeled as each other or as _adjective_. Among the tokens analyzed in the SPACY library, _adjective_ had the highest rate of mis-annotation at 24%. Among these mis-annotations, 28 tokens were mistakenly labeled as _nouns_ and 27 as _verbs_. Additionally, 18 tokens originally identified as _conjunctions_ were mislabeled as _prepositions_ and _pronouns_. Similar to NLTK, there were instances where _nouns_ and _verbs_ were incorrectly labeled as each other.