SlowGuess/ACL_2020_OCR huggingface dataset:
For reproducibility, we list all the key hyperparaters we use for each method in the Table A2.
We implement our Knowledge Incorporation Module based on DocReader implementations. Entity embeddings are pretrained using TransE (Bordes et al., 2013) with the dimension of 100. The hyperparameters are kept same as the DocReader. All the models are run on NVIDIA GeForce GTX 1080 GPUs. We save the best model (with the highest EM) on the dev set and use it for test set.
B Performance on Shorter Contexts
Using the entire clinical record as the context might be too long for models to capture sequential information. We also try to split the entire record into different sections (e.g., "medical history", "family history") based on some heuristic measures. Specifically, in order to split the clinical notes into sections, we notice that most sections begin with easily identifiable headers. To detect these headers we use a combination of heuristics such as whether the line contains colons, all uppercase formatting
Method	Hyper-parameters Setting
DocReader	epoch: 30; batch-size: 16; test-batch-size:16; dropout-rate:0.4; doc-layers: 3; question-layers: 3; grad-clipping: 10; tune-partial: 1000; max-len: 30; the others are set as default
BiDAF	init_lr: 0.001; batch-size:6; num_epochs: 2; cluster: True; len-opt: True; word_count_th: 10; char_count_th: 50; sent_size_th: 4000; num_sents_th: 500; ques_size_th: 30; word_size_th: 30; para_size_th: 4000; the others are set as default;
QANet	batch-size: 4; lr: 0.001; grad-clip: 5; use-ema: True; epoch: 30; para_limit: 4000; ques_limit: 30; ans_limit: 30; char_limit: 40; num-head:1; the others are set as defalut;
BERT-base BioBERT ClinicalBERT	train_batch_size: 6; learning_rate: 3e-5; num_train_epochs: 3.0; max_seq_length: 384; doc_stride: 128; the others are set as default;
Table A2: Hyperparameters settings for all the methods used in the experiments.
Dataset	Model	Dev	Test
EM	F1	EM	F1
medication	DocReader	32.19	76.21	33.45	77.08
ClinicalBERT	30.16	74.81	32.18	75.79
relation	DocReader	87.21	94.32	87.54	94.97
ClinicalBERT	85.46	93.92	85.67	93.14
Table A3: Performance of the two models on the shorter context setting.
or phrases found in a list of clinical headers taken from SecTag (Denny et al., 2009). We then select the section that contains the answer as the context ( $\sim$ 100 words avg). We select DocReader and ClinicalBERT as representative methods and re-run them on the modified shorter context. The results are shown in Table A3. The performance of the two models is improved compared with the performance of models built on the whole record (long context). However, ClinicalBERT still does not outperform DocReader in this setting, indicating that longer context may not explain why BERT models do not win on this dataset or that shortening context in a such manner might break long dependencies.
This experiment setting may also inspire future research on "Open Clinical Reading Comprehension". Given that patients often have multiple clini
<sup>1</sup>https://rajpurkar.github.io/SQuAD-explorer/
$^{2}$ https://github.com/allenai/bi-att-flow
<sup>3</sup>https://github.com/facebookresearch/DrQA
$^{4}$ https://github.com/BangLiu/QANet-PyTorch
<sup>5</sup>https://github.com/google-research/bert
$^{6}$ https://github.com/dmis-lab/biobert
<sup>7</sup>https://github.com/EmilyAlsentzer/clinicalBERT
Distribution of Easy/Hard Questions	Easy	Hard	Total
Medication Relation	33,037 (73.1%) 165,271 (91.1%)	12,155 (26.9%) 16,193 (8.9%)	45,192 (100%) 181,464 (100%)
Results	Model	Easy	Hard	Total
EM	F1	EM	F1	EM	F1
Medication	DocReader (Chen et al., 2017)	30.25	73.78	13.26	61.46	25.68	70.45
ClinicalBERT (Alsentzer et al., 2019)	28.25	72.02	12.64	60.98	24.06	69.05
Relation	DocReader (Chen et al., 2017)	87.66	95.39	79.85	89.62	86.94	94.85
ClinicalBERT (Alsentzer et al., 2019)	86.06	93.71	78.09	86.57	85.33	93.06
Table A4: Performance of DocReader and ClinicalBERT on the easy/hard questions split.
cal records, it may not be feasible to jointly use all of them as context for one question. Given multiple records for one patient (instead of just one) and a question, the model would first need to retrieve the most relevant paragraphs and do reading comprehension on each of them or find clever ways to merge them. Such a setting would be interesting for future CminiRC datasets to explore.
C Easy/Hard Questions Split
We partition the questions into Easy and Hard. Specifically, we first train a BiLSTM reader and do the prediction on the test set. We obtain the performance of each question template by averaging the performance of all the questions made by this template (such template and question mappings are included in the emrQA dataset). Question templates that obtain higher performance than the overall performance are labeled as "Easy" and "Hard" otherwise. Then we map the difficulty level of question templates back to each question. The reason why we focus on splitting on the question template level is that we can avoid some random noise (e.g., random errors produced by the model on some questions). Also, we release the difficulty level of each question template so that users can easily know which questions are easy or hard and do not need to run a base model to obtain such mappings again. Distributions of easy/hard questions and results of the two selected models are shown in Table A4.