# Ontology Attention Layer for Medical Named Entity Recognition
## 5 Experimental Results and Discussion  
### Performance over Different Pretrained Language Models as the Encoders  
We validated the effectiveness of the proposed method using different pretrained language models as the encoders. We tested our method with BERT, BERT-WWM, and RoBERTa. In detail, for the CMeEE dataset, which is in Chinese, we used \"bert-base-chinese\" ([[https://huggingface.co/bert-base-chinese](https://huggingface.co/bert-base-chinese)]([https://huggingface.co/bert-base-chinese](https://huggingface.co/bert-base-chinese)), accessed on 6 October 2023), \"chinese-bert-wwm-ext\" [[https://huggingface.co/hfl/chinese-bert-wwm-ext](https://huggingface.co/hfl/chinese-bert-wwm-ext)]([https://huggingface.co/hfl/chinese-bert-wwm-ext](https://huggingface.co/hfl/chinese-bert-wwm-ext)), accessed on 6 October 2023) [26; 27], and \"chinese-roberta-wvm-ext\" ([[https://huggingface.co/hfl/chinese-robe](https://huggingface.co/hfl/chinese-robe)]([https://huggingface.co/hfl/chinese-robe](https://huggingface.co/hfl/chinese-robe)), rta-wvm-ext-large, accessed on 6 October 2023). For NCBI disease, which is in English, we used \"bert-base-uncased\" ([[https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)]([https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)), accessed on 6 October 2023) [25] and roberta-base ([[https://huggingface.co/roberta-base](https://huggingface.co/roberta-base)]([https://huggingface.co/roberta-base](https://huggingface.co/roberta-base)), accessed on 6 October 2023) [28]. Because BERT-WWM is not for English, we did not use a BERT-WWM model for the NCBI disease dataset. To assess the impact of the proposed algorithm on model recognition efficiency, all models were trained and tested on the same dataset using identical hyperparameters. The hyperparameters are tuned by grid search on the BERT model. The training and testing data for the medical corpus were also extracted from the same dataset. We compare the test results achieved by our proposed method, those achieved by the language models with the conventional self-attention layer, and those achieved by the vanilla language models. We used micro F1 score for CMeEE and macro F1 for NCBI, respectively, as the metric. The results in CMeEE and NCBI are shown in Figures 3 and 4, respectively. For CMeEE, the proposed ontology attention improved efficiency from 0.1% to 0.5% compared to the conventional self-attention layer. Furthermore, compared to the vanilla models, the proposed method improved efficiency from 0.1% to 1.5%. For NCBI disease, the proposed method consistently outperformed self-attention. However, it failed to outperform the groups that used the plain base model. This outcome may be attributed to the fact that the ontology was designed for the CBLUE dataset in Chinese and was translated into English for the NCBI disease task by non-medical professionals. A more professional translation may yield better results. Table 3 shows some typical outputs for the NCBI disease dataset. All groups encountered challenges in the recognition of certain difficult named entities, such as \"T-PLL\" in data #11. In texts without disease mentions, the plain base model and the self-attention group tended to generate more false positives, while the proposed method showed a reduction in false positives, as seen in data #18 and #81. Notably, the proposed method demonstrated superior efficacy in the identification of long complex named entities, exemplified in data #94, #138, and #155. The results above indicate that the proposed ontology attention mechanism outperforms existing self-attention mechanisms in recognizing challenging medical named entities and reducing false positives. The integration of ontology enhances the model's comprehension of medical terms. Consequently, the model can better capture context and semantic information related to medical entities, recognizing more complex medical terms and mislabeling fewer non-medical terms.