SlowGuess/ACL_2020_OCR huggingface dataset:
Yuhao Zhang $^{1}$ , Derek Merck $^{2}$ , Emily Bao Tsai $^{1}$ , Christopher D. Manning $^{1}$ , Curtis P. Langlotz $^{1}$
$^{1}$ Stanford University  $^{2}$ University of Florida
{yuhaozhang, ebtsai, manning, langlotz}@stanford.edu derek.merck@ufl.edu
Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where factual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones.
Neural abstractive summarization systems aim at generating sentences which compress a document while preserving the key facts in it (Nallapati et al., 2016b; See et al., 2017; Chen and Bansal, 2018). These systems are potentially useful in many real-world applications. For example, Zhang et al. (2018) have shown that customized neural abstractive summarization models are able to generate radiology summary statements with high quality by summarizing textual findings written by radiologists. This task has significant clinical value because of its potential to accelerate the radiology workflow, reduce repetitive human labor, and improve clinical communications (Kahn Jr et al., 2009).
Background: radiographic examination of the chest. clinical history: 80 years of age, male ...
Findings: frontal radiograph of the chest demonstrates repositioning of the right atrial lead possibly into the ivc. ... a right apical pneumothorax can be seen from the image. moderate right and small left pleural effusions continue. no pulmonary edema is observed. heart size is upper limits of normal.
Human Summary: pneumothorax is seen. bilateral pleural effusions continue.
Summary A (ROUGE-L = 0.77): no pneumothorax is observed. bilateral pleural effusions continue.
Summary B (ROUGE-L = 0.44): pneumothorax is observed on radiograph. bilateral pleural effusions continue to be seen.
Figure 1: A (truncated) radiology report and summaries with their ROUGE-L scores. Compared to the human summary, Summary A has high textual overlap (i.e., ROUGE-L) but makes a factual error; Summary B has a lower ROUGE-L score but is factually correct.
However, while existing abstractive summarization models are optimized to generate summaries that highly overlap with human references (Paulus et al., 2018), this does not guarantee factually correct summaries, as shown in Figure 1. Therefore, maintaining factual correctness of the generated summaries remains a critical yet unsolved problem. For example, Zhang et al. (2018) found that about  $30\%$  of the outputs from a radiology summarization model contain factual errors or inconsistencies. This has made such a system unusable in practice, as factual correctness is critically important in this domain to prevent medical errors.
Existing attempts at improving the factual correctness of abstractive summarization models have seen very limited success. For example, Cao et al. (2017) augmented the attention mechanism of neural models with factual triples extracted with open information extraction systems; Falke et al. (2019)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108-5120 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
studied using natural language inference systems to rerank generated summaries based on their factual consistencies; Krysciński et al. (2019b) proposed to verify factual consistency of generated summaries with a weakly-supervised model. Despite these efforts, none of the existing work has focused explicitly on optimizing an abstractive summarization system with a correctness objective. As a result, even state-of-the-art systems trained with ample data still produce summaries with a substantial number of factual errors (Goodrich et al., 2019; Krysciński et al., 2019a).
In this work we aim to optimize the factual correctness of existing neural summarization systems, with a focus on summarizing radiology reports. This task has several key properties that make it ideal for studying factual correctness in summarization models. First, the clinical facts or observations present in radiology reports have less ambiguity compared to open-domain text, which allows objective comparison of facts. Second, radiology reports involve a relatively limited space of facts, which makes automatic measurement of factual correctness in the generated text approachable. Lastly, as factual correctness is a crucial metric in this domain, improving factual correctness will directly lead to an ability to use the system.
To this end, we design a framework where an external information extraction system is used to extract information in the generated summary and produce a factual accuracy score by comparing it against the human reference summary. We further develop a training strategy where we combine a factual correctness objective, a textual overlap objective and a language model objective, and jointly optimize them via reinforcement learning (RL).
On two datasets of radiology reports collected from different hospitals, we show that our training strategy substantially improves the factual correctness of the summaries generated by a competitive neural summarization system. Moreover, we observe for the first time that, even in the absence of a factual correctness objective, optimizing a textual overlap-based metric substantially improves the factual correctness of the resulting system compared to maximum likelihood training. We further show via human evaluation and analysis that our training strategy leads to summaries with higher overall quality and correctness and which are closer to the human-written ones.
Our main contributions are: (i) we propose a
general framework and a training strategy for improving the factual correctness of summarization models by optimizing a multi-part objective via RL; (ii) we apply the proposed strategy to radiology reports, and empirically show that it improves the factual correctness of the generated summaries; and (iii) we demonstrate via radiologist evaluation that our system is able to generate summaries with clinical validity close to human-written ones. To our knowledge, our work represents the first attempt at directly optimizing a neural summarization system with a factual correctness objective via RL.
Neural Summarization Systems. Neural models for text summarization can be broadly divided into extractive approaches (Cheng and Lapata, 2016; Nallapati et al., 2016a) and abstractive approaches (Nallapati et al., 2016b; See et al., 2017). While existing models are often trained in an end-to-end manner by maximizing the likelihood of the reference summaries, RL has been shown useful in recent work (Chen and Bansal, 2018; Dong et al., 2018). Specifically, Paulus et al. (2018) found that directly optimizing an abstractive summarization model on the ROUGE metric via RL can improve the summary ROUGE scores. Our work extends the rewards used in existing work with a factual correctness reward to further improve the correctness of the generated summaries.
Factual Correctness in Summarization. Our work is closely related to recent work that studies factual correctness in summarization. Cao et al. (2017) proposed to improve summarization models by attending to fact triples extracted using open information extraction systems. Goodrich et al. (2019) compared different information extraction systems to evaluate the factual accuracy of generated text. Falke et al. (2019) explored using natural language inference systems to evaluate the correctness of generated summaries, and found models trained on existing datasets to be inadequate. Krysciński et al. (2019b) proposed to evaluate factual consistencies in the generated summaries using a weakly-supervised fact verification model. Despite these efforts, none of this work has shown success in directly optimizing a summarization system for factual correctness, and to our knowledge our work represents the first attempt in this direction. While our work is focused on improving neural summarization models, we note that the idea of
using information extraction systems to evaluate the fidelity of generated text has also been explored for data-to-text generation (Wiseman et al., 2017; Dhingra et al., 2019).
Summarization of Radiology Reports. Zhang et al. (2018) first studied the problem of automatic generation of radiology impressions by summarizing textual radiology findings, and showed that an augmented pointer-generator model achieves high overlap with human references. MacAvaney et al. (2019) extended this model with an ontology-aware pointer-generator and showed improved summarization quality. Li et al. (2019) and Liu et al. (2019) studied generating textual descriptions of radiology findings from medical images, and proposed RL-based approaches to tackle this problem. While Zhang et al. (2018) found that about  $30\%$  of the radiology summaries generated from neural models contain factual errors, improving factual correctness in radiology summarization remains unstudied.
3 Task & Baseline Pointer-Generator
We start by briefly introducing the task of summarizing radiology findings. Given a passage of radiology findings represented as a sequence of tokens  $\mathbf{x} = \{x_{1}, x_{2}, \ldots, x_{N}\}$ , with  $N$  being the length of the findings, the task involves finding a sequence of tokens  $\mathbf{y} = \{y_{1}, y_{2}, \ldots, y_{L}\}$  that best summarizes the salient and clinically significant findings in  $\mathbf{x}$ . In routine radiology workflow, an output sequence  $\mathbf{y}$  is produced by the radiologist, which we treat as a reference summary sequence. $^{1}$
To model the summarization process, we use the background-augmented pointer-generator network (Zhang et al., 2018) as the backbone of our method. This abstractive summarization model extends a pointer-generator (See et al., 2017) with a separate background section encoder and is shown to be effective in summarizing radiology notes with multiple sections. We briefly describe this model and refer readers to the original papers for details.
At a high level, this model first encodes the input sequence  $\mathbf{x}$  into hidden states with a Bi-directional Long Short-Term Memory (Bi-LSTM) network, and then generates an output sequence  $\mathbf{y}$  with a separate LSTM decoder. To make the input information available at decoding time, an attention
mechanism (Bahdanau et al., 2015) over the input hidden states is also added to the decoder.
The baseline pointer-generator model by Zhang et al. (2018) adds two augmentations to this attentional encoder-decoder model to make it suitable for summarizing radiology findings:
Copy Mechanism. To enable the model to copy words from the input, a copy mechanism (Vinyals et al., 2015; See et al., 2017) is added to calculate a generation probability at each step of decoding. This generation probability is then used to blend the original output vocabulary distribution and a copy distribution to generate the next word.
Background-guided Decoding. As shown in Figure 1, radiology reports often consist of a background section which documents the crucial study background information (e.g., purpose of the study, patient conditions), and a findings section which documents clinical observations. While words can be copied from the findings section to form the summary, Zhang et al. (2018) found it worked better to separately encode the background section, and inject the representation into the decoding process by concatenating it with the input.
4 Fact Checking in Summarization
Summarization models such as the one described in Section 3 are commonly trained with the teacher-forcing algorithm (Williams and Zipser, 1989) by maximizing the likelihood of the reference, human-written summaries. However, this training strategy results in a significant discrepancy between what the model sees during training and test time, often referred to as the exposure bias issue (Ranzato et al., 2016), leading to degenerate output at test time.
An alternative training strategy is to directly optimize standard metrics such as ROUGE scores (Lin, 2004) with RL and this was shown to improve summarization quality (Paulus et al., 2018). Nevertheless, this method still provides no guarantee that the generated summary is factually accurate and complete, since the ROUGE scores merely measure the superficial text overlap between two sequences and do not account for the factual alignment between them. To illustrate this, a reference sentence pneumonia is seen and a generated sentence pneumonia is not seen have substantial text overlap and thus the generated sentence would achieve a high ROUGE score, however the generated sentence conveys an entirely opposite fact. In this section
<sup>1</sup>While the name "impression" is often used in clinical settings, we use "summary" and "impression" interchangeably.
Figure 2: Our proposed training strategy. Compared to existing work which relies only on a ROUGE reward  $r_{\mathrm{R}}$ , we add a factual correctness reward  $r_{\mathrm{C}}$  which is enabled by a fact extractor. The summarization model is updated via RL, using a combination of the NLL loss, a ROUGE-based loss and a factual correctness-based loss. For simplicity we only show a subset of the clinical variables in the fact vectors  $\mathbf{v}$  and  $\hat{\mathbf{v}}$ .
we first introduce a method to verify the factual correctness of the generated summary against the reference summary, and then describe a training strategy to directly optimize a factual correctness objective to improve summary quality.
4.1 Evaluating Factual Correctness via Fact Extraction
A convenient way to explicitly measure the factual correctness of a generated summary against the reference is to first extract and represent the facts in a structured format. To this end, we define a fact extractor to be an information extraction (IE) module, denoted as  $f$ , which takes in a summary sequence  $y$  and returns a structured fact vector  $\mathbf{v}$ :
$$
\mathbf {v} = f (y) = \left(v _ {1}, \dots , v _ {m}\right) \tag {1}
$$
where  $v_{i}$  is a categorical variable that we want to measure via fact checking and  $m$  the total number of such variables. For example, in the case of summarizing radiology reports,  $v_{i}$  can be a binary variable that describes whether an event or a disease such as pneumonia is present or not in a radiology study.
Given a fact vector  $\mathbf{v}$  output by  $f$  from a reference summary and  $\hat{\mathbf{v}}$  from a generated summary, we further define a factual accuracy score  $s$  to be the ratio of variables in  $\hat{\mathbf{v}}$  which equal the corresponding variables in  $\mathbf{v}$ , namely:
$$
s (\hat {\mathbf {v}}, \mathbf {v}) = \frac {\sum_ {i = 1} ^ {m} \mathbb {1} [ v _ {i} = \hat {v} _ {i} ]}{m} \tag {2}
$$
where  $s \in [0,1]$ . Note that this method requires a summary to be both precise and complete in order to achieve a high  $s$  score: missing out a positive variable or falsely claiming a negative variable will be equally penalized.
Our general definition of the fact extractor module  $f$  allows it to have different realizations for different domains. For our task of summarizing radiology findings, we make use of the open-source CheXpert radiology report labeler (Irvin et al., 2019). At its core, the CheXpert labeler parses the input sentences into dependency structures and runs a series of surface and syntactic rules to extract the presence status of 14 clinical observations seen in chest radiology reports. It was evaluated to have over  $95\%$  overall  $\mathrm{F}_1$  when compared against oracle annotations from multiple radiologists on a large-scale radiology report dataset.
4.2 Improving Factual Correctness via Policy Learning
The fact extractor module introduced above not only enables us to measure the factual accuracy of a generated summary, but also provides us with an opportunity to directly optimize the factual accuracy as an objective. This can be achieved by viewing our summarization model as an agent, the actions of which are to generate a sequence of words to form the summary  $\hat{y}$ , conditioned on the input  $x$ . The agent then receives rewards  $r(\hat{y})$  for its actions, where the rewards can be designed to measure the quality of the generated summary. Our goal is to learn an optimal policy  $P_{\theta}(y|x)$  for the summarization model, parameterized by the network parameters  $\theta$ , which achieves the highest expected reward under the training data.
Formally, we minimize loss  $\mathcal{L}$ , the negative ex
$^{2}$ https://github.com/stanfordmlgroup/chexpert-labeler
3For this study we used a subset of these variables and discuss the reasons in Appendix A.
4 For clarity, we drop the bold symbol and use  $x$  and  $y$  to represent the input and output sequences, respectively.
punctuation of the reward  $r(\hat{y})$  over the training data:
$$
\mathcal {L} (\theta) = - \mathbb {E} _ {\hat {y} \sim P _ {\theta} (y | x)} [ r (\hat {y}) ]. \tag {3}
$$
The gradient can be calculated as (REINFORCE Williams, 1992):
$$
\nabla_ {\theta} \mathcal {L} (\theta) = - \mathbb {E} _ {\hat {y} \sim P _ {\theta} (y | x)} [ \nabla_ {\theta} \log P _ {\theta} (\hat {y} | x) r (\hat {y}) ]. \tag {4}
$$
In practice, we approximate this gradient over a training example with a single Monte Carlo sample and deduct a baseline reward to reduce the variance of the gradient estimation:
$$
\nabla_ {\theta} \mathcal {L} (\theta) \approx - \nabla_ {\theta} \log P _ {\theta} (\hat {y} _ {s} | x) (r (\hat {y} _ {s}) - \bar {r}), (5)
$$
where  $\hat{y}_s$  is a sampled sequence from the model and  $\bar{r}$  a baseline reward. Here we adopt the self-critical training strategy (Rennie et al., 2017), where we obtain the baseline reward  $\bar{r}$  by applying the same reward function  $r$  to a greedily decoded sequence  $\hat{y}_g$ , i.e.,  $\bar{r} = r(\hat{y}_g)$ . We empirically find that using this self-critical baseline reward helps stabilize the training of our summarization model.
4.3 Reward Function
The learning strategy in Equation (5) provides us with the flexibility to optimize arbitrary reward functions. Here we decompose our reward function into two parts:
$$
r = \lambda_ {1} r _ {\mathrm {R}} + \lambda_ {2} r _ {\mathrm {C}}, \tag {6}
$$
where  $r_{\mathrm{R}} \in [0,1]$  is a ROUGE reward, namely the ROUGE-L score (Lin, 2004) of the predicted sequence  $\hat{y}$  against the reference  $y$ ;  $r_{\mathrm{C}} \in [0,1]$  is a correctness reward, namely the factual accuracy  $s$  of the predicted sequence against the reference sequence, as in Equation (2);  $\lambda_1, \lambda_2 \in [0,1]$  are scalar weights that control the balance between the two. To measure the similarity between the reference and the generation, we also experimented with more recent metrics that rely on neural representations of text, such as the BERTScore (Zhang et al., 2020). However, we found that these metrics, mostly trained on web and newswire data, generalize poorly to our domain of text.
Paulus et al. (2018) found that directly optimizing a reward function without the original negative log-likelihood (NLL) objective as used in teacher-forcing can hurt the readability of the generated summaries, and proposed to alleviate this problem by combining the NLL objective with the RL loss.
Number of Examples
Split	Stanford	RIH
Train	89,992 (68.8%)	84,194 (60.3%)
Dev	22,031 (16.8%)	25,966 (18.6%)
Test	18,827 (14.4%)	29,494 (21.1%)
Total	130,850	139,654
Table 1: Statistics of the Stanford and RIH datasets.
Here we adopt the same strategy, and our final loss during training is:
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {\mathrm {R}} + \lambda_ {2} \mathcal {L} _ {\mathrm {C}} + \lambda_ {3} \mathcal {L} _ {\mathrm {N L L}}, \tag {7}
$$
where  $\lambda_3\in [0,1]$  is an additional scalar that controls the weight of the NLL loss.
Our overall training strategy is illustrated in Figure 2. Our final loss jointly optimizes three aspects of the summaries:  $\mathcal{L}_{\mathrm{NLL}}$  serves as a conditional language model that optimizes the fluency and relevance of the generated summary,  $\mathcal{L}_{\mathrm{R}}$  controls the brevity of the summary and encourages summaries which have high overlap with human references, and  $\mathcal{L}_{\mathrm{C}}$  encourages summaries that are factually accurate when compared against human references.
We collected two real-world radiology report datasets and describe our experiments using them as our main training and evaluation corpora.
5.1 Data Collection
We collected anonymized chest radiographic reports within a certain period of time from two collaborating hospitals: the Stanford University Hospital and the Rhode Island Hospital (RIH).<sup>5</sup>
For both datasets, we ran simple preprocessing following Zhang et al. (2018). To test the generalizability of the models, instead of using random stratification, we stratified each dataset over time into training, dev and test splits. We include statistics of both datasets in Table 1 and preprocessing and stratification details in Appendix B.
As we use the augmented pointer-generator network described in Section 3 as the backbone of our method, we mainly compare against it as the
Our retrospective study has been approved by the corresponding institutional review boards with waiver of consent.
System	Stanford	RIH
R-1	R-2	R-L	Factual F1	R-1	R-2	R-L	Factual F1
LexRank (Erkan and Radev, 2004)	26.8	16.3	23.6	—	20.6	10.7	18.3	—
BanditSum (Dong et al., 2018)	32.7	20.9	29.0	—	26.1	14.0	23.3	—
PG Baseline (Zhang et al., 2018)	48.3	38.8	46.6	55.9	54.1	44.7	52.2	69.3
PG + RLr	52.0	41.1	49.5	63.2	58.0	47.2	55.7	73.3
PG + RLc	50.7	39.7	48.0	65.9	55.2	45.4	52.9	75.4
PG + RLr+C	52.0	41.0	49.3	64.5	57.0	46.6	54.7	74.8
Table 2: Main results on the two datasets. R-1, R-2, R-L represent the ROUGE scores. PG Baseline represents our baseline augmented pointer-generator;  $\mathrm{RL}_{\mathbb{R}}$ ,  $\mathrm{RL}_{\mathbb{C}}$  and  $\mathrm{RL}_{\mathbb{R} + \mathbb{C}}$  represent RL training with the ROUGE reward alone, with the factual correctness reward alone and with both. All the ROUGE scores have a  $95\%$  confidence interval of at most  $\pm 0.6$ .  $\mathrm{F}_1$  scores for extractive models were not evaluated for the reason discussed in Section 5.3.
baseline model (PG Baseline), and use the open implementation by Zhang et al. (2018).
For the proposed RL-based training, we compare three variants: training with only the ROUGE reward  $(\mathrm{RL}_{\mathrm{R}})$ , with only the factual correctness reward  $(\mathrm{RL}_{\mathrm{C}})$ , or with both  $(\mathrm{RL}_{\mathrm{R} + \mathrm{C}})$ . All three variants have the NLL component in the training loss as in Equation (7). For all variants, we initialize the model with the best baseline model trained with standard teacher-forcing, and then finetune it on the training data with the corresponding RL loss, until it reaches the best validation score.
To understand the difficulty of the task and evaluate the necessity of using abstractive summarization models, we additionally evaluate two extractive summarization methods: (1) LexRank (Erkan and Radev, 2004), a widely-used non-neural extractive summarization algorithm; and (2) BanditSum (Dong et al., 2018), a state-of-the-art RL-based neural extractive summarization model. For both methods we use their open implementations. We include other model implementation and training details in Appendix C.
5.3 Evaluation