## 5 Experiments
### Error Analysis  
#### 5.3.1 Quantitative Analysis  
Since our approaches to the proposed data involve two stages, i.e., document retrieval and reading comprehension, both stages could be the potential error sources. We first check the performance of the IR-based document retrieval stage by letting two medical doctors annotate whether the top 25 retrieved paragraphs by IR-Cust contain enough evidence for answering the questions over 100 randomly selected samples from the development set. We have three annotation levels: full evidence (evidence is totally complete to derive the answer), partial evidence (evidence is useful but not complete), and no evidence (no evidence is found at all). Table 9 summarizes the results of the annotation. We also summarize the percentages of samples for which we find full, partial, or no evidence individually for type 1 and type 2 questions on the MCMLE and TWMLE datasets. From this table, we see that only the MCMLE dataset has a good retrieval recall while for the majority of samples in USMLE, we cannot find any evidence in the retrieved text. Moreover, among those samples that can find full or partial evidence, we calculate the percentage of samples that find the evidence in the top \(N\) (1, 5, 10, 15) retrieved paragraphs, as shown in Table 10. Overall, we have two findings: (1). The MCMLE and TWMLE datasets have much higher percentage of type 1 questions than USMLE, which is positively correlated with their much better IR retrieval performance; (2). Such poor evidence retrieval performance for the USMLE dataset should be the main cause of the extremely low baseline models' performance as shown in Table 8 (neural models are even beaten by the IR baseline).  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Datasets** & \multicolumn{2}{c}{**USMLE**} & \multicolumn{2}{c}{**TWMLE**} \\  & **Dev** & **Test** & **Dev** & **Test** \\ \hline Chance & 25.0 & 25.0 & 25.0 & 25.0 \\ \hline PMI & 29.8 & 31.1 & 30.8 & 31.1 \\ IR-ES & 34.0 & 35.5 & 24.9 & 26.8 \\ IR-Custom & **38.3** & 36.1 & 35.1 & 34.8 \\ \hline Max-out & 28.9 & 28.6 & 29.4 & 27.8 \\ \hline BERT-Base-En & 33.9 & 34.3 & 34.3 & 33.3 \\ clinicalBERT-Base & 33.7 & 32.4 & 33.4 & 32.1 \\ BioRoBERTa-Base & 35.1 & 36.1 & 38.7 & 36.9 \\ BioBERT-Base & 34.3 & 34.1 & 41.6 & 41.1 \\ \hline RoBERTa-Large & 35.2 & 35.0 & 39.6 & 39.3 \\ BioBERT-Large & 36.1 & **36.7** & **42.2** & **42.0** \\ \hline \hline \end{tabular}
\end{table}
Table 8: Performance of baselines in accuracy (%) on the USMLE and TWMLE datasets.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Types** & **USMLE** & **MCMLE** & **TWMLE** \\ \hline Full Evidence & 24.0 & 75.0 (82.7/52.0) & 60.0 (63.4/52.6) \\ Partial Evidence & 8.0 & 21.0 (14.7/40.0) & 16.7 (19.5/10.5) \\ No Evidence & 68.0 & 4.0 (2.6/8.0) & 23.3 (17.1/36.9) \\ \hline \hline \end{tabular}
\end{table}
Table 9: Percentage of samples that can find full evidence, partial evidence, or no evidence in the top 25 retrieved paragraphs. For numbers within parenthesis, the left number is for type 1 while the right number is for type 2 questions. Since almost all questions are type 2 for USMLE, we do not have percentage numbers specifically for type 1 and 2 questions.