# Iterative Annotation of Biomedical NER Corpora with Deep Neural Networks and Knowledge Bases
## 4 Experimental Assessment and Discussion
### B-NER Annotated Corpora  
The original unannotated dataset is formed by the narrative parts of NL text extracted from a set of 1011 anonymized EHRs in the Italian language, which has a total word count of \(1,657,970\). In detail, the dataset contains EHRs acquired from the eHealth systems of some different hospitals in Italy. As mentioned above, the EHRs had been previously anonymized and they are related to patients admitted to different departments of the hospitals. The content of these documents is relatively homogeneous, containing the clinical diary of the patients, where the causes of the admission to the hospitals, the diseases, the prognosis, the follow-ups, the exams, the procedures and the prescriptions are described. Some sample sentences extracted from two different EHRs (translated into English) are reported below. * _Found sub-capital fracture and dislocation of left shoulder and contusion of right hip caused by accidental fall at home._
* _Tomorrow follow-up exams._
* _Patient admitted to cardiology from 9 February to 19 February due to episodes of arrhythmia, likely secondary to chronic renal failure._  
Eight different named-entity classes are identified, as shown in Table 1, following UMLS semantic types [52] and considering at the same time possible real-world applications of the trained ML models [2]. As explained in Section 3.1, in the preliminary step of the annotation process a small set of documents formed by the text extracted from 25 randomly selected EHRs was manually annotated by two domain experts. The annotation procedure was conducted according to predefined guidelines, which describe general and specific annotation rules. The labelling \begin{table}
\begin{tabular}{c c c} \hline \hline
**Class Type** & **Acronym** & **Examples** \\ \hline Diseases and Symptoms & DIS & Febbre (_Fever_), pressione alta (_High blood pressure_), citrosi \\  & & epatica (_liver cirrhosis_) \\ Drug names & DRU & Paracetamol (_Paracetamol_), antibiotico (_antibiotic_) \\ Departments & DEP & Ortopedia (_Orthopedics_), pronto soccoros (_emergency_) \\  & & Ecografo (_ultrasound scanner_, profilassi antithrombotica (_thrombosis prophylaxis_), stent (_stent_) \\  & & Piede destroy (_right foot_), testa \\ Body Parts & BOD & dell’omero (_lumen head_), \\ Measures & MEA & 30 cc, 12 mm, 120 bpm \\ Dates & DAT & 23 giugno 2012 (_23 June_ 2012), oggi (_today_), ore 12:30 (_12:30_) \\  & & Radiografa (_radiography_), \\ Diagnostic procedures or lab & & valutazione cardiologica \\ tests & ANA & (_cardiac assessment_), glicemia \\  & & (_glycaemia_), coronografia \\ _(angiography_) & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Entity classes with corresponding acronyms and examples. The English translation of the examples is in italics between parentheses. process followed the _IOB_ notation [53], i.e., each token belonging to an entity is labelled with the corresponding class adding the prefix \(B\) (Begin) if it is the first token of the entity, the prefix \(I\) for all subsequent tokens of the same multi-word entity and the tag \(O\) (Outside) if the token does not belong to an entity. The result of the manual annotation is a small dataset, which counts 7421 tokens and 1963 named entities, as shown in the first row of Table 2. The experts worked for approximately eight hours to produce this dataset, including the discussion about conflicts and disambiguation of the conflicting annotations. To the end of providing a stop criterion for the iterative AL phase (see Section 3.1), a further test set, which counts 21,133 tokens, was also manually annotated. This small dataset is used to train the DNN Bi-LSTM-CRF [43] architecture. This DL model has been used to automatically annotate new documents randomly extracted from the whole dataset, starting the iterative AL phase. In each iteration, the human experts had to review the correctness of the annotations produced by the DL model, eventually correcting the wrong or the missing ones.