# An Integrated Approach to Biomedical Term Identification Systems
## 5 Evaluation
### Error Analysis  
The main purpose of this section is to carry out an error analysis to identify the weaknesses of our system. To this end, we have obtained some basic statistics for the first 2000 documents of the MANTRA subset used:  
* The average number of entities annotated in the MANTRA Medline corpus is 4.2 per document. * The average number of entities recognized by the BSB system is 5.8 concepts per document. * The average number of entities recognized by FreelingMed is 4.6 per document. We have tried to identify which entities are not recognized by the BSB system. For this, we have taken 200 random documents with 483 medical entities and characterized the concepts annotated in MANTRA Medline that BSB was unable to detect. Table 2 summarizes the situations where concepts were not detected by the BSB system and the main cause. \begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **Precision** & **Recall** & **F1** & **Accuracy** \\ \hline BSB & 0.5651 & 0.7694 & 0.5466 & 0.4101 \\ FreelingMed & 0.5419 & 0.5314 & 0.4091 & 0.2961 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Results obtained on the MANTRA Medline corpus. Below are several examples of missing terms in BSB. In each example, the text of the document is shown with the unidentified entity marked in bold, the CUI of UMLS as annotated in MANTRA and, finally, the atoms that we have found for that CUI in our UMLS dictionary in Spanish. * Missing plurals: Because our system does not recognize plurals, it cannot match between the word \"fluctuaciones\" and \"Fluctuacion\", so they are not annotated. In Table 3 we can see this example. * False verbs: As can be seen in Table 4, our system detected the entity \"Estudio\" at first match, but the POS Tagger analysis returned the _verb_ POS for the word \"Estudio\", so it was discarded. * Genre distinction: BSB does not differentiate between masculine and feminine, therefore it is unable to match \"Analgesica\" and \"Analgesico\". This example is shown in Table 5.  
* Abbreviations: In Table 6 we can see an example of abbreviations. BSB does not find the concordance between \"TRH\" and \"Treorina\" because it does not process any acronyms. \begin{table}
\begin{tabular}{l l} \hline Text: & **Estudio** comparative de las fluctuaciones cclicas de glucosa, insulin y glucagón en el plasma sanguineo de macacos, papiones y humano en ayunas. \\ \hline MANTRA matching: & [C0557651] **Estudio** \\ \hline Spanish atoms: & **Estudio** \\ \hline \end{tabular}
\end{table} 
Table 4: False verbs example. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Error** & **Percentage** & **Number of Errors** \\ \hline Plurals & 10\% & 16 \\ False verbs & 8\% & 12 \\ Distinct Genre & 1\% & 2 \\ Abbreviations & 1\% & 2 \\ UMLS Code unknown & 13\% & 20 \\ MANTRA errors & 10\% & 15 \\ Other reasons & 57\% & 88 \\ Total & 100\% & 155 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: BSB classification errors. * Unknown UMLS codes: There are cases in which MANTRA corpus recognizes UMLS codes and our system does not have them in its dictionary, an example of these codes are: C0024554, C0031843 or C0243107. * Annotation errors: In MANTRA corpus there are documents with invalid annotated concepts so we do not find any relationship between the term identified and its meaning in UMLS. An example of this case can be seen in Table 7.
* Other reasons: In Table 8 we can observe that in MANTRA corpus the entity \"water\" is annotated with the code C1550678, but in Spanish UMLS that code contains the atoms \"Especimen de agua\" and \"Muestra de agua\". Our system fails to detect \"agua\" because \"agua\" is a term that is within an atom of UMLS is not an atom by itself. To finish this study we wanted to perform an analysis of the groups of entities that are classified in MANTRA Medline and how our system fails to address them.