SlowGuess/ACL_2020_OCR huggingface dataset:
We use two sets of metrics to evaluate model performance at the corpus level. First, we use the standard ROUGE scores (Lin, 2004), and report the  $\mathrm{F_1}$  scores for ROUGE-1, ROUGE-2 and ROUGE-L, which compare the word-level unigram, bigram and longest common sequence overlap with the reference summary, respectively.
For factual correctness evaluation, we use a Factual  $\mathbf{F}_1$  score. While the factual accuracy score  $s$  that we use in the reward function evaluates how factually accurate a specific summary is, compar-
ing it at the corpus level can be misleading, for the same reason that accuracy is a misleading measure in information retrieval (Manning et al., 2008). To understand this, imagine the case where a clinical variable  $v$  has rare presence in the corpus. A model which always generates a negative summary for it (i.e.,  $v = 0$ ; the disease is not present) can have high accuracy, but is useless in practice. Instead, for each variable, we obtain a model's predictions over all test examples and calculate its  $\mathrm{F}_1$  score. We then macro-average the  $\mathrm{F}_1$  of all variables to obtain the overall factual  $\mathrm{F}_1$  score of the model.
Note that the CheXpert labeler that we use is specifically designed to run on radiology summaries, which usually have a different style of language compared to the radiology findings section of the reports (see further analysis in Section 7). As a result, we found the labeler to be less accurate when applied to the findings section. For this reason, we were not able to estimate the factual  $\mathrm{F_1}$  scores on the summaries generated by the two extractive summarization models.
We first present our automatic evaluation results on the two collected datasets. We then present a human evaluation with board-certified radiologists where we compare the summaries generated by humans, the baseline and our proposed model.
Our main results on both datasets are shown in Table 2. We first notice that while the neural extractive model, BanditSum, outperforms the non-neural extractive method on ROUGE scores, our PG baseline model substantially outperforms both of them,
Variable	PG Baseline	RLR+C	Δ
No Finding	77.3	81.5	+4.2*
Cardiomegaly	29.5	40.4	+10.9*
Airspace Opacity	64.6	74.9	+10.3*
Edema	58.4	70.9	+12.5*
Consolidation	46.3	53.2	+6.9*
Pneumonia	46.7	46.8	+0.2
Atelectasis	48.8	56.3	+7.5*
Pneumothorax	69.5	82.9	+13.4*
Pleural Effusion	62.0	73.4	+11.4*
Macro Avg.	55.9	64.5	+8.6*
suggesting that on both datasets abstractive summarization is necessary to generate summaries comparable to human-written ones. We further show that this difference is likely due to the different styles of language (see Section 7): while radiologists tend to use more compressed language when writing the summaries, extractive methods produce more verbose summaries that fail to capture this difference.
On the Stanford dataset, training the pointer-generator model with ROUGE reward alone  $(\mathrm{RL}_{\mathbb{R}})$  leads to improvements on all ROUGE scores, with a gain of 2.9 ROUGE-L scores. Training with the factual correctness reward alone  $(\mathrm{RL}_{\mathrm{C}})$  leads to the best overall factual  $\mathrm{F_1}$  with a substantial gain of  $10\%$  absolute, however with consistent decline in the ROUGE scores compared to  $\mathrm{RL}_{\mathbb{R}}$  training. Combining the ROUGE and the factual correctness rewards  $(\mathrm{RL}_{\mathrm{R} + \mathrm{C}})$  achieves a balance between the two, leading to an overall improvement of 2.7 on ROUGE-L and  $8.6\%$  on factual  $\mathrm{F_1}$  compared to the baseline. This indicates that  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  training leads to both higher overlap with references and improved factual correctness.
Most surprisingly, while ROUGE has been criticized for its poor correlation with human judgment of quality and insufficiency for evaluating correctness of the generated text (Chaganty et al., 2018), we find that optimizing ROUGE reward jointly with NLL leads to substantially more factually correct summaries than the baseline, shown by the notable gain of  $7.3\%$  factual  $\mathrm{F}_1$  from the  $\mathrm{RL}_{\mathrm{R}}$  training.
All of our findings are consistent on the RIH dataset, with  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  achieving an overall improve
Table 3: Test set factual  $\mathrm{F}_1$  scores for each variable on the Stanford dataset. * marks statistically significant improvements with  $p < .01$  under a bootstrap test.
Stanford Dataset
Background: radiographic examination of the chest ...Findings: continuous rhythm monitoring device again seen projecting over the left heart. persistent low lung volumes with unchanged cardiomegaly. again seen is a diffuse reticular pattern with interstitial prominence demonstrated represent underlying emphysematous changes with superimposed increasing moderate pulmonary edema. small bilateral pleural effusions. persistent bibasilar opacities left greater than right which may represent infection versus atelectasis.
Human: increased moderate pulmonary edema with small bilateral pleural effusions. left greater than right basilar opacities which may represent infection versus atelectasis.
PG Baseline (s = 0.33): no significant interval change.
RLR+C (s = 1.00): increasing moderate pulmonary edema. small bilateral pleural effusions. persistent bibasilar opacities left greater than right which may represent infection versus atelectasis.
RIH Dataset
Background: history: lobar pneumonia, unspecified organism ...Findings: lines/tubes: none. lungs: right middle lobe airspace disease seen on prior radiographs from <date> and <date> is no longer evident. bilateral lungs appear clear. pleura: there is no pleural effusion or pneumothorax. heart and mediastinum: no cardiomegaly. thoracic aorta appears calcified and mildly tortuous. bones: ...
Human: no acute cardiopulmonary abnormality.
PG Baseline (s = 0.75): right middle lobe airspace disease could represent atelectasis, aspiration or pneumonia.
RLR+C (s = 1.00): no acute cardiopulmonary abnormality.
Figure 3: Truncated examples from the test sets along with human, PG baseline and  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  outputs. Factual accuracy scores (s) are also shown for the model outputs. For the Stanford example, clinical observations in the summaries are marked for clarity; for RIH, a wrongly copied observation is marked.
ment of 2.5 ROUGE-L and  $5.5\%$  factual  $\mathrm{F_1}$  scores.
Fine-grained Correctness. To understand how improvements in individual variables contribute to the overall improvement, we show the fine-grained factual  $\mathrm{F}_1$  scores for all variables on the Stanford dataset in Table 3 and include results on the RIH dataset in Appendix D. We find that on both datasets, improvements in  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  can be observed on all variables tested. We further find that, as we change the initialization across different training runs, while the overall improvement on factual  $\mathrm{F}_1$  stays approximately unchanged, the distribution of the improvement on different variables can vary substantially. Developing a training strategy for fine-grained control over different variables is an interesting direction for future work.
Qualitative Results. In Figure 3 we present two example reports along with the human references, the PG baseline outputs and  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  outputs. In the first example, while baseline output seems generic and does not include any meaningful observation, the summary from the  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  model aligns well with the reference, and therefore achieves a higher
Metric	Win	Tie	Lose
Our Model vs. PG Baseline
Fluency	7%	60%	33%
Factual Correctness	31%	55%	14%
Overall Quality	48%	24%	28%
Our Model vs. Human Reference
Fluency	17%	54%	29%
Factual Correctness	23%	49%	28%
Overall Quality	44%	17%	39%
factual accuracy score. In the second example, the baseline model wrongly copied an observation from the findings although the actual context is no longer evident, while the  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  model correctly recognizes this and produces a better summary.
To study whether the improvements in the factual correctness scores lead to improvement in summarization quality under expert judgment, we run a comparative human evaluation following previous work (Chen and Bansal, 2018; Dong et al., 2018; Zhang et al., 2018). We sampled 50 test examples from the Stanford dataset, and for each example we presented to two board-certified radiologists the full radiology findings along with blinded summaries from (1) the human reference, (2) the PG baseline and (3) our  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  model. We shuffled the three summaries such that the correspondence cannot be guessed, and asked the radiologists to compare them based on the following three metrics: (1) fluency, (2) factual correctness and completeness, and (3) overall quality. For each metric we asked the radiologists to rank the three summaries, with ties allowed. After the evaluation, we converted each ranking into two binary comparisons: (1) our model versus the baseline model, and (2) our model versus human reference.
The results are shown in Table 4. Comparing our model against the baseline model, we find that: (1) in terms of fluency our model is less preferred, although a majority of the results  $(60\%)$  are ties; (2) our model wins more on factual correctness and overall quality. Comparing our model against
Table 4: Results of the radiologist evaluation. The top three rows present results when comparing our  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  model output versus the baseline model output; the bottom three rows present results when comparing our model output versus the human-written summaries.
System	Stanford pplx.	RIH pplx.
Human	6.7	5.5
LexRank	10.8	36.9
BanditSum	9.9	40.9
PG Baseline	4.8	3.8
PG + RLr+C	6.5	4.8
Table 5: Perplexity scores as evaluated by the trained radiology impression LM on the test set human references and model predictions.
human references, we find that: (1) human wins more on fluency; (2) factual correctness results are close, with  $72\%$  of our model outputs being at least as good as human; (3) surprisingly, in terms of overall quality our model was slightly preferred by the radiologists compared to human references. Lastly, when comparing the baseline model against human references, we find that outputs from the baseline model are much less correct and lower-quality than human summaries.
7 Analysis & Discussion
Fluency and Style of Summaries. Our human evaluation results in Section 6.2 suggest that in terms of fluency our model output is less preferred than human reference and baseline output. To further understand the fluency and style of summaries from different models at a larger scale, we trained a neural language model (LM) for radiology summaries following previous work (Liu et al., 2018). Intuitively, radiology summaries which are more fluent and consistent with humans in style should be able to achieve a lower perplexity under this in-domain LM, and vice versa. To this end, we collected all human-written summaries from the training and dev split of both datasets, which in total gives us about 222,000 summaries. We then trained a strong Mixture of Softmaxes LM (Yang et al., 2018) on this corpus, and evaluated the perplexity of test set outputs for all models.
The results are shown in Table 5. We find that while extractive models can achieve non-trivial overlap with references, their perplexity scores tend to be much higher than humans. We conjecture that this is because radiologists are trained to write the summaries with more compressed language than when they are writing the findings, therefore sentences directly extracted from the findings tend to be more verbose than needed.
Figure 4: Distributions of the top 10 most frequent trigrams from model outputs on the Stanford test set.
We further observe that the baseline model achieves even lower perplexity than humans, and our proposed method leads to a perplexity score much closer to human references. We hypothesize that this is because models trained with teacher-forcing are prone to generic generations which are fluent and relevant but may not be factually correct. Training with the proposed rewards alleviates this issue, leading to summaries more consistent with humans in style. For example, we find that no significant interval change is a very frequent generation from the baseline, regardless of the actual input. This sentence occurs in  $34\%$  of the baseline outputs on the Stanford dev set, while the number for  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  and human are only  $24\%$  and  $17\%$ . This hypothesis is further confirmed when we plot the distribution of the top 10 most frequent trigrams from different models in Figure 4: while the baseline heavily reuses the few most frequent trigrams, our model  $\mathrm{RL}_{\mathrm{R} + \mathrm{C}}$  tends to have more diverse summaries which are closer to human references. The same trends are observed for 4-grams and 5-grams.
Limitations. While we showed the success of our proposed method on improving the factual correctness of a radiology summarization model, we also recognize several limitations of our work. First, our proposed training strategy crucially depends on the availability of an external IE module. While this IE module is relatively easy to implement for a domain with a limited space of facts, how to generalize this method to open-domain summarization remains unsolved. Second, our study was based on a rule-based IE system, and the use of a more robust statistical IE model can potentially improve the results. Third, we mainly focus on key factual errors which result in a flip of the binary outcome of an event (e.g., presence of disease), whereas factual errors in generated summaries can occur in other forms such as wrong adjectives or
coreference errors (Krysciński et al., 2019a). We leave the study of these problems to future work.
In this work we presented a general framework and a training strategy to improve the factual correctness of neural abstractive summarization models. We applied this approach to the summarization of radiology reports, and showed its success via both automatic and human evaluation on two separate datasets collected from hospitals.
Our general takeaways include: (1) in a domain with a limited space of facts such as radiology reports, a carefully implemented IE system can be used to improve the factual correctness of neural summarization models via RL; (2) even in the absence of a reliable IE system, optimizing the ROUGE metrics via RL can substantially improve the factual correctness of the generated summaries.
We hope that our work draws the community's attention to the factual correctness issue of abstractive summarization models and inspires future work in this direction.
The authors would like to thank the anonymous reviewers, Peng Qi and Urvashi Khandelwal for their helpful comments, and Dr. Jonathan Movson for his help with obtaining the RIH data used in this study.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. The 2015 International Conference on Learning Representations.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017. Faithful to the original: Fact aware neural abstractive summarization. The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-2017).
Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018).
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018).
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit: Sum: Extractive summarization as a contextual bandit. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018).
Günes Erkan and Dragomir R Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457-479.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 19).
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. 2019. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019).
Charles E Kahn Jr, Curtis P Langlotz, Elizabeth S Burnside, John A Carrino, David S Channin, David M Hovsepian, and Daniel L Rubin. 2009. Toward best practices in radiology reporting. *Radiology*, 252(3):852-856.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In The 2015 International Conference for Learning Representations.
Wojciech Krysciński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019).
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840.
Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2019. Hybrid retrieval-generation reinforced agent for medical image report generation. Advances in neural information processing systems.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out: ACL Workshop.
Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically accurate chest X-ray report generation. arXiv preprint arXiv:1904.02633.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. The 2018 International Conference for Learning Representations.
Sean MacAvaney, Sajad Sotudeh, Arman Cohan, Nazli Goharian, Ish Talati, and Ross W. Filice. 2019. Ontology-aware clinical abstractive summarization. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 19).
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge University Press.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-2017).
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016b. Abstractive text summarization using sequence-to-sequence RNNs and beyond. Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. The 2016 International Conference on Learning Representations.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In The 2017 Annual Meeting of the Association of Computational Linguistics (ACL 2017).
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256.
Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. *Neural Computation*, 1(2):270-280.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017).
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. The 2018 International Conference for Learning Representations.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In The 2020 International Conference for Learning Representations.
Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. 2018. Learning to summarize radiology findings. In EMNLP 2018 Workshop on Health Text Mining and Information Analysis.
Time Coverage
Split	Stanford	RIH
Train	2009/01 – 2014/04	2017/11 – 2018/06
Dev	2014/05 – 2014/08	2018/07 – 2018/09
Test	2014/09 – 2014/12	2018/10 – 2018/12
Table 6: Time coverage of different splits in the Stanford and RIH datasets.
A Clinical Variables Inclusion Criteria
While the CheXpert labeler that we use is able to extract status for 14 clinical variables, we found that several variables are very rarely represented in our corpora and therefore using all of them makes the calculation of the factual  $\mathrm{F_1}$  score very unstable. For example, we found that training the same model using different random initializations would result in highly varying  $\mathrm{F_1}$  scores for these variables. For this reason, for both datasets we removed from the factual  $\mathrm{F_1}$  calculation all variables which have less than  $3\%$  positive occurrences on the validation set. We further removed the variables "Pleural Other" and "Support Devices" due to their ambiguity. This process results in a total of 9 variables for the Stanford dataset and 8 for the RIH dataset.