# Iterative Annotation of Biomedical NER Corpora with Deep Neural Networks and Knowledge Bases
## 4 Experimental Assessment and Discussion
### Performances
After the expansion and the balancing of the training set using the second part of the proposed approach, where new sentences are obtained leveraging DS with domain KBs containing lists of entities of two more imbalanced classes, the performance of the ELMo DNN trained on the training set obtained with both the AL and DS phases are sensibly improved, as shown in Table 8. In this case, we reported only the results obtained by the best performing model, which was the Bi-LSTM CRF architecture with the ELMo embeddings. This behaviour is expected, due to overfitting issues of the BERT model trained on very large datasets [55]. In particular, comparing the obtained results for DRU and DEP classes in Table 8, where the DS augmentation for balancing and expansion were applied for the annotation of the training set after the AL, with the results achieved in the same class types shown in Table 7, where only the AL is performed, it is possible to observe that the DS augmentation applied to the most unbalanced classes DEP and DRU provided a sensible performance boost. Moreover, we can also note an improvement in all the other classes thanks to the oversampling performed during the DS data augmentation. To the end of further verifying the effectiveness of the final obtained annotated corpus, we also tested the DNN models on the out-of-corpus test set, previously described in Section 4.1. This additional manually annotated test set was extracted from a different document collection, which contains many entities not present in the dataset used to build and annotate the training set. Table 9 shows the results obtained by the ELMo BiLSTM CRF architecture trained on the final annotated corpus and tested on the out-of-corpus test set. It is worth noting that, despite a slight performance drop, the DNN model still performs at a good level, assessing the effectiveness of the obtained annotated training corpus. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Entity Type** & **Precision** & **Recall** & **F1-Score** \\ \hline MEA & 0.8436 & 0.8599 & 0.8517 \\ DRU & 0.8085 & 0.3576 & 0.4959 \\ DEP & 0.1845 & 0.1404 & 0.1595 \\ THE & 0.5668 & 0.8459 & 0.6788 \\ BOD & 0.8283 & 0.8949 & 0.8603 \\ DIS & 0.8316 & 0.9125 & 0.8702 \\ DAT & 0.8905 & 0.9492 & 0.9189 \\ ANA & 0.8145 & 0.9137 & 0.8612 \\ \hline Average & 0.7635 & 0.7889 & 0.7760 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Performance of the ELMo Bi-LSTM CRF at the last step of the AL phase of the annotation procedure, in terms of precision, recall and F1-Score for each class. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Entity Type** & **Precision** & **Recall** & **F1-Score** \\ \hline MEA & 0.9636 & 0.9675 & 0.9655 \\ DRU & 0.9863 & 0.9893 & 0.9878 \\ DEP & 0.9878 & 0.9860 & 0.9869 \\ THE & 0.9609 & 0.9636 & 0.9622 \\ BOD & 0.9203 & 0.9262 & 0.9232 \\ DIS & 0.9595 & 0.9634 & 0.9615 \\ DAT & 0.9783 & 0.9809 & 0.9796 \\ ANA & 0.9647 & 0.9718 & 0.9682 \\ \hline Average & 0.9642 & 0.9679 & 0.9661 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Results obtained by the ELMo Bi-LSTM CRF trained with the final annotated corpus (AL and DS augmentation) in terms of precision, recall and F1-Score for each class. In summary, these results demonstrate that the DS data augmentation phase is capable of further improving the quality of the dataset obtained from the previous iterative AL phase, mitigating the issues of the AL related to unbalanced classes and out-of-corpus named entities. Finally, the next Table 10 reports the metrics averaged on all classes obtained by each considered DNN model, namely the Bi-LSTM CRF with the various considered WE models as input layer and the fine-tuned BERT model, trained on the final annotated dataset (AL and DS) and tested on the out-of-corpus test set. The purpose of this last experiment is to evaluate the contribution of different neural language models on a corpus containing many named entities not present in the training set.