# Automatic Extraction of Adverse Drug Reactions from Summary of Product Characteristics
## 3 Results
### Error Analysis  
Three main types of errors are identified among the 116 incorrect extracted ADRs of Reviewer 1 (false positives). Table 4 summarizes the error analysis of incorrectly extracted ADRs. Our approach encountered issues with splitting multiple ADR terms that are joined by a colon or semicolon. For instance, two ADR terms (agitation and aggression) could be extracted from 'Agitation, aggression'. However, if multiple ADR terms are joined by 'or', 'and', or semicolon, such as the examples shown in Table 4, our extraction method fails to properly address this. The second type of false-positive results is related to the data cleaning of our approach. The extracted ADRs contain noises such as unrelated words or special characters. As shown in the examples, to get the correct ADRs, we need to clean up noises like '\(\backslash\)u2020', and 'in combination with insulin or sulphonylurea'. Lastly, a small number of extracted terms are not ADRs at all. For example,'skin and subcutaneous tissue disorders' is an SoC term that should be excluded from our final ADR lists. Since other incorrect extractions do not belong to any specific group, it is difficult to exclude them. However, the number of such errors is so small that its impact is limited.  
Since there are only seven missing ADRs out of 1706 extracted ADRs, false-negative (missing ADR) is not a common error. However, we found that such an error is more likely to happen to medicines using free text to describe their ADRs. Since only 87 out of the selected 647 medicines summarize their ADRs in free-text, and the number of unstructured reported medicines has been steadily decreasing with recent SmPC updates,  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Type of Error**} & \multicolumn{3}{c}{**Examples**} & \multirow{2}{*}{**Counts**} \\ \cline{2-2} \cline{4-4}  & **Extracted Terms** & & **Correct ADRs** \\ \hline \multirow{2}{*}{Unsplit multiple ADTs} & “angio-oedema or & angio-oedema, & \\  & anaphylactic reactions” & anaphylactic reactions & \\ \hline \multirow{4}{*}{ADR with noise} & - & ’retinal & \\  & & detachment\(\backslash\)u2020’ & \\ \cline{1-1}  & - & ’hypoglycaemia in & retinal detachment, \\ \cline{1-1}  & & combination with & hypoglycaemia & \\ \cline{1-1}  & & insulin or & \\ \cline{1-1}  & & sulphonylurea’ & \\ \hline \multirow{4}{*}{Non-ADR} & - & ’all causality & \\ \cline{1-1}  & & frequency’, & \\ \cline{1-1}  & - & ’general disorders.’ & \\ \cline{1-1}  & - & ’skin and & Not ADR & 13 \\ \cline{1-1}  & & subcutaneous tissue & \\ \cline{1-1}  & & disorders’ & \\ \hline \hline \end{tabular}
\end{table}
Table 4: Error analysis for false positives.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Reviewer 1** & **Reviewer 2** & \\  & **(NLP Expert)** & **(Clinical Expert)** & **Totals** \\ \hline \# of reviewed medicines & 32 & 5 & 37 \\ \hline \# of extracted ADRs terms & 1700 & 118 & 1824 \\ \hline \# of correct ADRs (TP) & 1590 & 110 & 1703 \\ \hline \# of incorrect ADRs (FP) & 116 & 8 & 124 \\ \hline \# of missing ADRs (FN) & 7 & 11 & 18 \\ \hline Recall & 0.996 & 0.909 & 0.99 \\ \hline Precision & 0.932 & 0.932 & 0.932 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Final results of the manual expert reviews.  
the false-negative error has a relatively limited and decreasing impact on the performance of our approach.