## 3 Data
### Data Collection
#### 3.2.2 Document Collection  
Extensive medical knowledge is needed to answer every question in our MedQA data. For people to obtain answers for these questions, they need to obtain necessary knowledge from a volume of medical textbooks during years of training. Similarly, for a machine learning model to be successful in this task, we need to grant it access to the same collection of text materials as human have. Therefore, for USMLE, we prepared text materials from a total of 18 English medical textbooks that have been widely used by medical students and USMLE takers, whereas for MCMLE, we collected 33 simplified Chinese medical textbooks designated as the official textbooks for preparing the medical licensing exam in Mainland China. For TWMLE, since medical students in Taiwan use the same textbooks as those in USA for exam preparation, USMLE and TWMLE would use the same document collection for solving questions. We will release the textbooks we collected upon the license agreement of research use only.  
All textbooks we collected are originally in PDF format and we converted them into digital text via OCR. We performed some clean-up pre-processing over the converted text such as misspelling correction, and then divided all text into paragraphs. Table 4 summarizes the statistics of the document collection.  
To evaluate whether these collected text materials can cover enough knowledge to answer those questions, we randomly extracted 100 questions from the development set of all three datasets and let two medical experts with the MD degree annotate how many of them can be answered by the evidence from our prepared text materials. Table 5 summarizes the results of this evaluation. From this table, we see that our collected text materials can provide enough information for answering most of the questions in our data.  
\begin{table}
\begin{tabular}{l r r} \hline \hline
**Metric** & **USMLE/TWMLE** & **MCMLE** \\ \hline \# of books & 18 & 33 \\ \# of paragraphs & 231,581 & 116,216 \\ \# of tokens & 12,727,711 & 14,730,364 \\ Vocabulary size & 245,851 & 4695 \\ Avg./Max. paragraph length & 55.0/1234 & 126.7/9082 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Overall statistics of the document collection. USMLE and TWMLE share the same document collection. Token number and vocabulary size are counted in tokens for English and in characters for Chinese.  
\begin{table}
\begin{tabular}{l r r r} \hline \hline  & **USMLE** & **MCMLE** & **TWMLE** \\ \hline Number of options per question & 4 & 4 & 4 \\ Avg./Max. option len. & 3.5/45 & 7.3/100 & 20.6/210 \\ Avg./Max. question len. & 116.6/530 & 45.7/333 & 61.0/1950 \\ vocab/character size & 63,317 & 3263 & 3588 \\ \hline \hline \multicolumn{4}{l}{**Number of questions**} \\ Train & 10,178 & 27,400 & 11,298 \\ Development & 1272 & 3425 & 1412 \\ Test & 1273 & 3426 & 1413 \\ All & 12,723 & 34,251 & 14,123 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Overall statistics of MedQA. Question/option length and vocabulary/character size are calculated in tokens for English and in characters for Chinese. Vocabulary/character size is measured on the combination of questions and options. “Avg./Max. option len.” represents “Average/Maximum option length”.