## 3 Data
### Data Analysis  
Based on a preliminary analysis of our proposed data, we find that it poses unique challenges for language understanding compared with existing OpenQA datasets, elaborated below:  
* **Professional Knowledge:** For most existing QA datasets, the question answering process relies largely on the basic understanding of language and general world knowledge reasoning. Some works have revealed that the large-scale pretrained language models bear a certain level of common-sense and symbolic reasoning capability besides their linguistic knowledge [32; 33], which may significantly contribute to the remarkable performance of current QA models. However, the answering of every question in our dataset needs abundant professional domain-specific knowledge, particularly medical knowledge, which forces the model to have a deep understanding of the extracted context.
* **Diversity of Questions:** The field of clinical medicine is diverse enough for questions to be asked about a wide range of topics. In general, there are two categories of questions: (1). The question is asking for a single piece of knowledge, for instance, via the question \"Which of the following symptoms belongs to schizophrenia?\" (2). The question first describes a patient's condition and then asks for _the most probable diagnosis/the most appropriate treatment/the examination needed/the mechanism of certain conditions/the possible outcome for a certain treatment_, etc. Table 1 shows two typical examples for type 2. Table 6 summarizes the percentages of these two types of questions for each dataset by annotating randomly selected 100 questions from the development set. Typically, type 1 questions need one-step reasoning while type 2 questions require multi-hop reasoning and are thus much more complicated than type 1 ones, imposing challenges not only to the reading comprehension model but also to the relevant text retrieval module. For example, in order to solve the first question in Table 1, the model needs to first extract, understand, and interpret the symptoms of this patient among a long paragraph of description, then match these symptoms to millions of medical knowledge text snippets and find out the most relevant one, and finally understand the evidence sentence for answer selection.
* **Complex Reasoning over Multiple Evidence:** Many questions in our data involve complex multi-hop reasoning over several evidence snippets. For instance, the second example is a typical question that requires multiple steps of reasoning over three evidences, where from the symptoms and signs we can know that this patient is highly likely an ALS case by looking at the evidence 1 and 2. Afterwards, from evidence 3, we know that the SOD1 is the possible genetic mutation for familial ALS, which is the correct answer.
* **Noisy Evidence Retrieval:** Retrieving relevant information from large-scale text is much more challenging than reading a short piece of text. Passages from textbooks often do not directly give answers to questions and many passages retrieved by the most widely adopted term-matching based information retrieval (IR) systems turn out to be noisy distractors and not relevant, especially for type 2 questions. For those questions involving multi-hop reasoning, models must identify all relevant information scattered in different passages, and missing any single piece of evidence would lead to failures.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **USMLE** & **MCMLE** & **TWME** \\ \hline Coverage (\%) & 88.0 & 100.0 & 87.0 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Percentage of questions that human experts can find enough evidence from our collected text material to obtain correct answers for by randomly annotating 100 samples from the development set.