# Information Extraction from German Clinical Care Documents in Context of Alzheimer's Disease
## 3 Materials and Methods
### Medical Report Pipeline
#### 3.2.2 Inter-Annotator Agreement  
For the five mentioned deficits and their onset, we determined the inter-annotator agreement (IAA), as this helps to analyze the complexity of the wording that can be used by different physicians. For this purpose, two field experts annotated a total of 25 medical report documents. The F1-measure was used to determine the IAA, using the following formula where _TP_, _FP_ and _FN_ stand for _true positive, false positive_ and _false negative_, respectively. \[precision=\frac{TP}{TP+FP} \tag{1}\]  
\[recall=\frac{TP}{TP+FN} \tag{2}\]  
\[\text{F1-score}=2\times(\frac{precision\times recall}{precision+recall}) \tag{3}\]  
However, before annotating the final IAA document set, three annotation rounds--each with ten documents--were performed. After each round, the annotation guidelines were adapted in a consecutive discussion. To determine the IAA, we used the annotations of one annotator as the gold standard and compared them to the results of the second annotator. The results of the IAA can be seen in Table 4. We assessed the results of cognitive deficits extraction at the sentence as well as the document level. The medical experts classified the deficits as being approved or negated. Thus, the absolute amount of negated as well as approved entities was calculated. However, at the document level, in some cases, contradictory information might be detected by the system. To cope with such cases, we performed a post-processing step in which the entity with the highest amount was taken into account. For example, if a memory deficit was detected twice as being approved and once as being negated, the approved entity was taken into account for the evaluation. If the occurrence of both cases was equally high, the approved one was taken into account. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Section** & **Entity Class** & **Example** & **No. of Instances** \\ \hline Personal info. & Date of visit & berichten über... am DD\_MM\_YYYY & 49 \\ Diagnosis & Diagnosis & alzheimer-Krankheit und subkotrikale & 84 \\  & & arterisklerotische Encephalopathic & 84 \\ Examination summary & MMSE score & Mini-Mental-Status v. 29.10.2007: & 100 \\  & & 24 von 30 Punkten & 25 \\ Examination summary & MMSE date & Mini-Mental-Status v. 29.10.2007: 24 von 30 & 25 \\  & & Punkten & \\ Anamnesis & Onset time & und berichtete, seit ca. 6 Monaten vergesslich & 74 \\  & & und unkonzentriert zu sein. & \\ Anamnesis & Memory deficit & und berichtete, seit ca. 6 Monaten vergesslich & 168/07 * \\  & & & \\ Anamnesis & Language deficit & mache umstandliche Bescheribungen wegen & 44/12 * \\  & & teils Worrfindungssfourungen & \\ Anamnesis & Attention deficit & Krozentration beientrchtigt, Auffassung und & 16/07 * \\  & & Merkfähigkeit waren eingeschrankt. & \\  & & beispielsweise bei komplexeren Tattigkeiten & \\ Anamnesis & Planning deficit & wie dem Regeln von Finanzangelegenheiten & 25/17 * \\  & & gebe es zunehmed Unsicherheiten & \\ Anamnesis & Orientation deficit & Keine raumlichen Orientierungssfourungen & 34/22 * \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Overview of all annotated classes subject to the extraction from medical reports. The amount column refers to the number of instances of the manually annotated test set (on sentence level). * The amounts are split into approved and negated (app./neg.) statements (on sentence basis). These post-processing steps were applied on both the manually curated and automatically extracted data. We also chose to not evaluate the negation detection separately, but take only the final result into account, so that we were able to judge on the final performance of the developed systems. Except for the attention deficit, the IAA of all classes is higher on document than on sentence level. The highest agreement is achieved for the memory deficit and amounts to an F1-score of 86% on sentence level and 97% on document level. For language and orientation deficits as well as the date of the disease onset, an agreement above 90% on document level is achieved.