# Natural Language Processing in Knowledge-Based Support for Operator Assistance
## 4 Result and Discussion
Furthermore, 22 _nouns_ were mistakenly labeled as _pronouns_ and 26 as _adjective_. For Stanford, the highest mis-annotation was 38%, which occurred for _conjunctions_, mostly mis-annotated as _prepositions_ and _adverbs_ as for NLTK. A total of 30 _adjective_ tokens were mistakenly annotated as _nouns_ and 24 _adjective_ tokens were mistakenly annotated as _verbs_. In addition 24 out of 30 mis-annotated _verbs_ were identified as _nouns_ in Stanford taggers. Upon closer examination of mis-annotations, distinct patterns emerged in the labeling behavior of NLTK, SpaCy, and Stanford NLP. Within NLTK, the most prevalent mislabeling occurrences involved _conjunctions_ being incorrectly labeled as _prepositions_, _pronouns_ mistakenly identified as _determiners_, _conjunctions_ labeled as _adverbs_, and _adjective_ mislabeled as _nouns_. In the case of SpaCy, a notable tendency was observed in misclassifying _adjective_ as _nouns_ and _adjectives_ as _verbs_. Stanford NLP exhibited its own set of common mislabeling instances, including _adjectives_ being miscategorized as _nouns_ and _verbs_, _conjunctions_ mislabeled as _adverbs_ and _prepositions_, _nouns_ inaccurately tagged as _adjective_ and _verbs_, and _verbs_ erroneously identified as _nouns_. To assess the classification of declarative and imperative sentences, we designated the manual category as declarative, while the warning, informative, and work-instruction categories were categorized as imperative. We evaluated the performance of each library separately for these two sentence types across the eight tags. The NLTK library attained an accuracy of 91% for declarative sentences and 89% for imperative sentences. Similarly, SpaCy exhibited accuracies of 91% for declarative sentences and 95% for imperative sentences. Additionally, Stanford achieved 90% accuracy for declarative sentences and 93% for imperative sentences. Figure 6 illustrates a comparison between NLTK, SPACY, and Stanford in terms of imperative and declarative sentences. In the case of imperative sentences, NLTK exhibited lower accuracy for _adverbs_ and _conjunctions_ compared to other tag types. Conversely, SPACY performed well across all tag types for imperative sentences. In the Stanford library, for the imperative sentences, _conjunctions_ were less accurately tagged compared to other tags. \begin{table}
\begin{tabular}{l c c c} \hline \hline  & **NLTK** & **SPACY** & **Stanford** \\ \hline Adjective & 18.0\% & 22.0\% & 22.0\% \\ Adverb & 10.0\% & 4.0\% & 4.0\% \\ Conjunction & 38.0\% & 12.0\% & 38.0\% \\ Determiner & 0.2\% & 1.0\% & 0.8\% \\ Noun & 9.0\% & 8.0\% & 6.0\% \\ Preposition & 2.0\% & 2.0\% & 4.0\% \\ Pronoun & 18.0\% & 1.0\% & 18.0\% \\ Verb & 12.0\% & 5.0\% & 6.0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Mis-annotations of token types in NLTK, SpaCy, and Stanford libraries. Further analysis revealed the percentage of mis-annotations for each tag in imperative and declarative sentences, as depicted in Figure 7 for each respective library. The confusion matrix presented in Figure 7 highlights the predominant types of mis-annotations observed in NLTK for declarative sentences, which include _pronouns_ mislabeled as _determiners_, _adjectives_ mislabeled as _nouns_, _conjunctions_ mislabeled as _adverbs_, and _conjunctions_ mislabeled as _prepositions_. Similarly, for imperative sentences in NLTK, the prevalent mislabeled tags encompassed _conjunctions_ mislabeled as _prepositions_, _adverbs_ mislabeled as _nouns_, _verbs_ mislabeled as _nouns_, and _adjectives_ mislabeled as _nouns_. In the case of SpaCy, there were no mis-annotations exceeding 10% for imperative sentences, while for declarative sentences, the prominent mislabeled tags consisted of _adjectives_ mislabeled as _nouns_, _adjectives_ mislabeled as _verbs_, and _conjunctions_ mislabeled as _pronouns_. For the Stanford taggers, in declarative sentences, the mislabel happened in _adjectives_ to _nouns_, _conjunctions_ to _adverbs_, _determiners_, and _prepositions_, and finally, _pronouns_ to _determiners_.