# Iterative Annotation of Biomedical NER Corpora with Deep Neural Networks and Knowledge Bases
## 5 Conclusions  
This paper presented an approach based on both active learning and distant supervision, which makes the manual annotation of a corpus for biomedical named entity recognition (B-NER) a less costly process, reducing the efforts needed by human experts. In detail, the method is based on a first AL phase, where a DNN architecture for NER composed of a BiLSTM-CRF is used to support the manual annotation. When no further improvements are achieved by the AL-based process, the corpus is augmented using DS, exploiting domain KBs, in order to mitigate the class imbalance. Finally, an assessment of the utility of using a WE model trained on a closed domain document collection as input \begin{table}
\begin{tabular}{c c c c} \hline \hline
**WE Model** & **Precision** & **Recall** & **F1-Score** \\ \hline _W2V ISTI_ & 0.7794 & 0.7714 & 0.7714 \\ _W2V cbow_ & 0.8047 & 0.8000 & 0.8010 \\ _W2V skip_ & 0.8676 & 0.8464 & 0.8545 \\ _FT cbow_ & 0.8164 & 0.8143 & 0.8125 \\ _FT skip_ & 0.8367 & 0.8107 & 0.8213 \\ _ELMo_ & **0.8809** & **0.8986** & **0.8875** \\ _BERT_ & 0.7356 & 0.7246 & 0.7301 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Results in terms of F1-Score, precision and recall obtained by the DNN on the out-of-corpus test set, using different pretrained WE models. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Entity Type** & **Precision** & **Recall** & **F1-Score** \\ \hline MEA & 0.9374 & 0.9051 & 0.9210 \\ DRU & 0.6429 & 0.8824 & 0.7438 \\ DEP & 0.9000 & 1.000 & 0.9474 \\ THE & 0.7983 & 0.8559 & 0.8261 \\ BOD & 0.9112 & 0.8701 & 0.8902 \\ DIS & 0.8475 & 0.9278 & 0.8858 \\ DAT & 0.5608 & 0.9222 & 0.6975 \\ ANA & 0.9364 & 0.8805 & 0.9076 \\ \hline Average & 0.8809 & 0.8986 & 0.8875 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Results in terms of precision, recall and F1-Score averaged on all classes obtained with the DNN with ELMo embeddings trained on the final annotated corpus and tested on the out-of-corpus test set. for the DNN was carried out, considering word2 vec, FastText and ELMo embeddings, and also comparing the obtained results with the fine tuned BERT model pretrained on a very large general domain document collection. The approach was tested by creating an Italian language B-NER corpus used to train different B-NER DNNs. The experiments demonstrated that the obtained corpus is capable of training a B-NER DNN with very good performance, allowing one to annotate an NER corpus in a fraction of the time required for a fully manual annotation. Moreover, they showed that the pretraining of the ELMo contextual embedding model on a biomedical closed domain corpus allows one to obtain results comparable with the more complex BERT architecture pretrained on a very large general domain document collection, which demands more computational resources. The proposed annotation methodology can facilitate the development and the implementation of AI-powered information extraction and indexing systems, improving the management of large natural language document collections, as well as supporting the analysis and the extraction of knowledge from such documents. On the other hand, a limit of the proposed approach is that KBs in the domain and the language of the annotations must be available to apply the DS phase. Moreover, the method is not fully automatic, requiring in any case human supervision, as well as a fully manual annotation in the preliminary phase. It also requires the availability of DL-dedicated hardware to carry out the AL phase in a reasonable time. Finally, the training of the NLM (in particular, the BERT-based models) requires the collection of a very large closed-domain unannotated document corpus, which in some cases may not be easy to obtain. In future work, the B-NER DL model trained on the obtained annotated corpus on more out-of-corpus documents, such as medical tweets or scientific papers, assessing the effectiveness of the proposed annotation methodology will be evaluated.