# Document Retrieval System for Biomedical Question Answering
## 2 Materials and Methods
### QA System Components
#### 2.2.3 Answer Extraction
As the first method, the F1 score of each feature was used in calculating their weight value. Hence, the F1 score of each feature (sentence, ner, cui, semgroup, semtype) was calculated separately. Fifty-seven sample questions were used for this evaluation. Tables 6 and 7 present the Top 10 and Top 20 evaluation results, respectively. The weight values were calculated by normalizing the weighted averages of the calculated F1 scores to 1. Tables 8 and 9 show the calculated weight scores. The final weight score was calculated using the arithmetic mean of the top10 and top20 evaluation weights. The final weight is presented in Table 10. \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Precision** & **Recall** & **F1-Score** \\ \hline Sim\({}_{\text{sentence}}\)(q,s) & 0.325974658869396 & 0.279680589455593 & 0.265812642336408 \\ Sim\({}_{\text{cui}}\)(q,s) & 0.342272347535505 & 0.281147607835646 & 0.281588644081949 \\ Sim\({}_{\text{nerr}}\)(q,s) & 0.212858535226956 & 0.191798144966908 & 0.172229828518593 \\ Sim\({}_{\text{semgroup}}\)(q,s) & 0.215350877192982 & 0.203630576246875 & 0.188343480739234 \\ Sim\({}_{\text{semtype}}\)(q,s) & 0.292933723196881 & 0.249584375421712 & 0.240911648270888 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Top 10 evaluation results of each feature. \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Precision** & **Recall** & **F1-Score** \\ \hline Sim\({}_{\text{sentence}}\)(q,s) & 0.289389042782395 & 0.424083519409149 & 0.309619576459582 \\ Sim\({}_{\text{cui}}\)(q,s) & 0.302934938705673 & 0.39415030999297 & 0.319192207483652 \\ Sim\({}_{\text{nerr}}\)(q,s) & 0.201702918113998 & 0.3084490564532 & 0.220466137401454 \\ Sim\({}_{\text{semgroup}}\)(q,s) & 0.211441982148353 & 0.338489889693882 & 0.236935883722674 \\ Sim\({}_{\text{semtype}}\)(q,s) & 0.252875243664717 & 0.396181088045895 & 0.280014580422107 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Top 20 evaluation results of each feature. \begin{table}
\begin{tabular}{l l l} \hline \hline  & **F1-Score** & **Weight** \\ \hline Sim\({}_{\text{sentence}}\)(q,s) & 0.265812642336408 & 0.231365501795192 \\ Sim\({}_{\text{cui}}\)(q,s) & 0.281588644081949 & 0.245097062973383 \\ Sim\({}_{\text{nerr}}\)(q,s) & 0.172229828518593 & 0.14991025388805 \\ Sim\({}_{\text{semgroup}}\)(q,s) & 0.188343480739234 & 0.163935708806267 \\ Sim\({}_{\text{semtype}}\)(q,s) & 0.240911648270888 & 0.209691472537108 \\  & Total & 1.14888624394707 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Top 10 normalized weights of each characteristic. \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Precision** & **Recall** & **F1-Score** \\ \hline Sim\({}_{\text{sentence}}\)(q,s) & 0.289389042782395 & 0.424083519409149 & 0.309619576459582 \\ Sim\({}_{\text{cui}}\)(q,s) & 0.302934938705673 & 0.39415030999297 & 0.319192207483652 \\ Sim\({}_{\text{nerr}}\)(q,s) & 0.201702918113998 & 0.3084490564532 & 0.220466137401454 \\ Sim\({}_{\text{semgroup}}\)(q,s) & 0.211441982148353 & 0.338489889693882 & 0.236935883722674 \\ Sim\({}_{\text{semtype}}\)(q,s) & 0.252875243664717 & 0.396181088045895 & 0.280014580422107 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Top 10 normalized weights of each characteristic. \begin{table}
\begin{tabular}{l l l} \hline \hline  & **F1-Score** & **Weight** \\ \hline Sim\({}_{\text{sentence}}\)(q,s) & 0.309619576459582 & 0.226623586325691 \\ Sim\({}_{\text{cui}}\)(q,s) & 0.319192207483652 & 0.233630197464604 \\ Sim\({}_{\text{nerr}}\)(q,s) & 0.220466137401454 & 0.16136843571909 \\ Sim\({}_{\text{semgroup}}\)(q,s) & 0.236935883722674 & 0.173423335541216 \\ Sim\({}_{\text{semtype}}\)(q,s) & 0.280014580422107 & 0.2049544444949398 \\  & Total & 1.36622838548947 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Top 20 normalized weights of each characteristic. The same method was used as the second weight calculation method. As an exception, two categories with low performances were not included in the calculation.