SlowGuess/ACL_2020_OCR huggingface dataset:
Additionally, apart from the positive and negative status, the CheXpert labeler is also able to generate an uncertain status for a variable, capturing observations with uncertainty, such as in the sentence "pneumonia is likely represented". While we can modify the factual accuracy score to take uncertainty into account, for simplicity in this work we do not make the distinction between a positive status and an uncertain status.
B Dataset Preprocessing and Stratification Details
We preprocessed both the Stanford and the RIH datasets following Zhang et al. (2018). All reports were first tokenized with Stanford CoreNLP (Manning et al., 2014). We then filtered the datasets by excluding reports where (1) no findings or impression (i.e., summary) section can be found; (2) multiple findings or impression sections can be found but cannot be aligned; or (3) the findings have fewer than 10 words or the impression has fewer than 2 words. Lastly, we replaced all date and time mentions with special tokens (e.g., <DATE>).
For both datasets, we stratified them over time
Variable	PG Baseline	RLR+C	Δ
No Finding	91.0	92.0	+1.0*
Cardiomegaly	21.1	33.8	+12.7*
Airspace Opacity	80.4	83.5	+3.1*
Edema	73.4	80.2	+6.8*
Pneumonia	63.5	69.2	+5.7*
Atelectasis	60.5	66.5	+6.0*
Pneumothorax	89.7	93.2	+3.5*
Pleural Effusion	74.3	79.9	+5.6*
Macro Avg.	69.3	74.8	+5.5*
Table 7: Test set performance for each variable on the RIH dataset. All numbers are  $\mathrm{F}_1$  scores. * marks statistically significant improvements with  $p < .01$  under a bootstrap test.
into training, dev and test splits. We employed this stratification strategy to test whether our model generalizes to future data when trained on historical data. We show the time coverage of each split in Table 6.
C Model Implementation and Training Details
For the baseline background-augmented pointer-generator model, we use its open implementation. We use a 2-layer LSTM as the findings encoder, 1-layer LSTM as the background encoder, and a 1-layer LSTM as the decoder. For all LSTMs we use a hidden size of 200. For the embedding layer we use 100-dimensional GloVe vectors (Pennington et al., 2014) which we pretrained on about 4 million radiology reports. We apply dropout (Srivastava et al., 2014) with  $p = 0.5$  to the embeddings. At decoding time, we use the standard beam search with a beam size of 5 and a maximum decoding length of 50.
For the training and finetuning of the models, we use the Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of  $1e^{-3}$ . We use a batch size of 64 and clip the gradient with a norm of 5. During training we evaluate the model on the dev set every 500 steps and decay the learning rate by 0.5 whenever the validation score does not increase after 2500 steps. Since we want the model outputs to have both high overlap with the human references and high factual correctness, for training we always use the average of the dev ROUGE score
$^{6}$ https://github.com/yuhaozhang/ summarize-radiology-findings
Stanford Dataset
Background: radiographic examination of the chest:  $<\text{date}>$ $<\text{time}>$ . clinical history:  $<\text{age}>$  years of age, with concern for pulmonary edema. procedure comments: 3 single views of the chest...
Findings: in the first chest radiograph from  $<\text{date}>$  at  $<\text{time}>$  there is interval intubation. Left arm-picc line remains in place, grossly unchanged persistent cardiomegaly, bilateral pleural effusion, and mild pulmonary edema. severe djd of the left gh joint is noted. In the second chest radiograph there is interval placement of a trialysis catheter in the left ij. no other significant changes are noted. In the third chest radiograph from  $<\text{date}>$  at  $<\text{time}>$  there is an increased left basilar opacity likely reflecting basilar consolidation, atelectasis or aspiration.
Human: in the final chest radiograph there is increased left basilar opacity likely reflecting basilar consolidation, atelectasis or aspiration.
PG Baseline: interval intubation with placement of a trialysis catheter in the left ij. grossly unchanged cardiomegaly, bilateral pleural effusion, and mild pulmonary edema.
$\mathbf{R}\mathbf{L}_{\mathbf{R} + \mathbf{C}}$  : interval placement of a trialysis catheter in the left ij. an increased left basilar opacity likely reflecting basilar consolidation, atelectasis or aspiration or aspiration.
RIH Dataset
Background: post op cardiac surgery - check lines and tubes. technique: single view of the chest obtained at  $<\text{time}>$ $<\text{date}>$  ...
Findings: lines/tubes: right ij sheath with central venous catheter tip overlying thesvc.on initial radiograph, endotracheal tube between the clavicular heads, and enteric tube with side port at the ge junction and tip below the diaphragm off the field-of-view; these are removed on subsequent film. mediastinal drains and left thoracostomy tube are unchanged. lungs: low lung volumes. retrocardiac airspace disease, slightly increased on most recent film. pleura: small left pleural effusion. no pneumothorax. heart and mediastinum: postsurgical widening of the cardiomeiastinal silhouette. aortic arch calcification. bones: intact median sternotomy wires.
Human: left basilar airspace disease and small left pleural effusion. lines and tubes positioned as above.
PG Baseline: lines and tubes as above. retrocardiac airspace disease, which may represent atelectasis, aspiration, or pneumonia.
$\mathbf{RL}_{\mathbb{R} + \mathbb{C}}$  : lines and tubes as described above. retrocardiac airspace disease, slightly increased on most recent film. small left pleural effusion.
Figure 5: More examples from the test splits of both datasets along with human, PG baseline and  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  summaries. In the first example, the baseline output successfully copied content from the context, but missed important observations. In the second example, the baseline output included some spurious facts that were not mentioned, and again neglected some important observations. In neither examples the  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  outputs make perfect summaries, but they represent better summaries than the baseline outputs.
and the dev factual  $\mathbf{F}_1$  score as the stopping criteria. We tune the scalar weights in the loss function on the dev sets and use weights of  $\lambda_{1} = 0.97$ ,  $\lambda_{2} = 0.97$  and  $\lambda_{3} = 0.03$  for both datasets.
For the extractive LexRank and BanditSum models, we use their open implementations. For the BanditSum extractive summarization model, we use default values for all hyperparameters as in Dong et al. (2018). For both models we select the top 3 scored sentences to form the summary, which yields the highest ROUGE-L scores on the dev sets.
For ROUGE evaluation, we use the Python
ROUGE implementation released by Google Research. We empirically find it to provide very close results to the original Perl ROUGE implementation by Lin (2004).
D Fine-grained Correctness Results on the RIH Dataset
We show the fine-grained factual  $\mathbf{F}_1$  scores for all variables on the RIH dataset in Table 7.
E More Examples with Baseline and System Generations
In Figure 5 we present more examples from both datasets along with the generations from the baseline system and our approach.
<sup>7</sup>https://github.com/miso-belica/sumy; <sup>8</sup>https://github.com/yuedongP/BanditSum
$^{8}$ https://github.com/google-research/google-research/tree/master/rouge