SlowGuess/NAACL_2021_OCR huggingface dataset:
We used the training and validation sets of MIMIC-CXR (Johnson et al., 2019) to train and validate models. MIMIC-CXR is a large publicly available database of chest radiographs. We extracted the findings sections from the reports with a text extraction tool for MIMIC-CXR $^2$ , and used them as our reference reports as in previous work (Liu et al., 2019; Boag et al., 2020). Findings section is a natural language description of the important aspects in a radiology image. The reports with empty findings sections were discarded, resulting in 152173 and 1196 reports for the training and validation set, respectively. We used the test set of MIMIC-CXR and the entire Open-i Chest X-ray dataset (Demner-Fushman et al., 2012) as two individual test sets. Open-i is another publicly available database of chest radiographs which has been widely used in past studies. We again extracted the findings sections, resulting in 2347 reports for MIMIC-CXR and 3335 reports for Open-i. Open-i is used only for testing since the number of reports is too small to train and test a neural report generation model.
BLEU4, CIDEr-D & BERTScore: We first use general NLG metrics to evaluate the generation quality. These metrics include the 4-gram BLEU score (Papineni et al., 2002, BLEU4), CIDEr score (Vedantam et al., 2015) with gaming penalties (CIDEr-D), and the  $\mathrm{F}_1$  score of the BERTScore (Zhang et al., 2020a).
Clinical Metrics: However, NLG metrics such as BLEU and CIDEr are known to be inadequate for evaluating factual completeness and consistency. We therefore followed previous work (Liu et al., 2019; Boag et al., 2020; Chen et al., 2020) by additionally evaluating the clinical accuracy of the generated reports using a clinical information extraction system. We use CheXbert (Smit et al., 2020), an information extraction system for chest reports, to extract the presence status of a series of observations (i.e., whether a disease is present or not), and score a generation by comparing the values of these observations to those obtained from
the reference<sup>3</sup>. The micro average of accuracy, precision, recall, and  $\mathrm{F}_1$  scores are calculated over 5 observations (following previous work (Irvin et al., 2019)) for: atelectasis, cardiomegaly, consolidation, edema, and pleural effusion<sup>4</sup>.
factENT & factENTNLI: We additionally include our proposed rewards factENT and factENTNLI as metrics to compare their values for different models.
We used  $\mathcal{M}^2$  Trans as our report generation model and used DenseNet-121 (Huang et al., 2017) as our image encoder. We trained  $\mathcal{M}^2$  Trans with the following variety of joint losses.
NLL  $\mathcal{M}^2$  Trans simply optimized with NLL loss as a baseline loss.
$\mathbf{NLL} + \mathbf{CDr}$  CIDEr-D and NLL loss is jointly optimized with  $\lambda_{1} = 0.01$  and  $\lambda_{2} = 0.99$  for the scaling factors.
NLL+BS The  $\mathrm{F}_1$  score of BERTScore and NLL loss is jointly optimized with  $\lambda_1 = 0.01$  and  $\lambda_2 = 0.99$ .
$\mathbf{NLL} + \mathbf{BS} + \mathbf{fc_E}$  factENT is added to NLL+BS with  $\lambda_{1} = 0.01$ $\lambda_{2} = 0.495$  , and  $\lambda_3 = 0.495$
$\mathbf{NLL} + \mathbf{BS} + \mathbf{fc}_{\mathbf{EN}}$  factENTNLI is added to NLL+BS with  $\lambda_{1} = 0.01$ ,  $\lambda_{2} = 0.495$ , and  $\lambda_{3} = 0.495$ .
We additionally prepared three previous models that have been tested on MIMIC-CXR.
TieNet We reimplemented the model of Wang et al. (2018) consisting of a CNN encoder and an RNN decoder optimized with a multitask setting of language generation and image classification.
CNN-RNN<sup>2</sup> We reimplemented the model of Liu et al. (2019) consisting of a CNN encoder and a hierarchical RNN decoder optimized with CIDEr and Clinically Coherent Reward
<sup>3</sup>We used CheXbert instead of CheXpert (Irvin et al., 2019) since CheXbert was evaluated to be approximately  $5.5\%$  more accurate than CheXpert. The evaluation using CheXpert can be found in Appendix C.
4These 5 observations are evaluated to be most represented in real-world radiology reports and therefore using these 5 observations (and excluding others) leads to less variance and more statistical strength in the results. We include the detailed results of the clinical metrics in Appendix C for completeness.
$^{2}$ https://github.com/MIT-LCP/mimic-cxr/tree/master/txt
Dataset	Model	NLG Metrics	Clinical Metrics (micro-avg)	Factual Rewards
BL4	CDr	BS	P	R	\( F_1 \)	acc.	\( f_{\text{E}} \)	\( f_{\text{EN}} \)
MIMIC-CXR	Previous models
TieNet (Wang et al., 2018)	8.1	37.2	49.2	38.6	20.9	27.1	74.0	-	-
CNN-RNN2(Liu et al., 2019)	7.6	44.7	41.2	66.4	18.7	29.2	79.0	-	-
R2Gen (Chen et al., 2020)	8.6	40.6	50.8	41.2	29.8	34.6	73.9	-	-
Proposed approach without proposed optimization
\( \mathcal{M}^2 Trans w/ NLL \)	10.5	44.5	51.2	48.9	41.1	44.7	76.5	27.3	24.4
\( \mathcal{M}^2 Trans w/ NLL+CDr \)	13.3	67.0	55.9	50.0	51.3	50.6	76.9	35.2	32.9
Proposed approach
\( \mathcal{M}^2 Trans w/ NLL+BS \)	12.2	58.4	58.4	46.3	67.5	54.9	74.4	35.9	33.0
\( \mathcal{M}^2 Trans w/ NLL+BS+fc_E \)	11.1	49.2	57.2	46.3	73.2	56.7	74.2	39.5	34.8
\( \mathcal{M}^2 Trans w/ NLL+BS+fc_{EN} \)	11.4	50.9	56.9	50.3	65.1	56.7	77.1	38.5	37.9
Open-i	Previous models
TieNet (Wang et al., 2018)	9.0	65.7	56.1	46.9	15.9	23.7	96.0	-	-
CNN-RNN2(Liu et al., 2019)	12.1	87.2	57.1	55.1	7.5	13.2	96.1	-	-
R2Gen (Chen et al., 2020)	6.7	61.4	53.8	27.0	17.3	21.1	94.9	-	-
Proposed approach without proposed optimization
\( \mathcal{M}^2 Trans w/ NLL \)	8.2	64.4	53.1	44.7	32.7	37.8	95.8	31.1	34.1
\( \mathcal{M}^2 Trans w/ NLL+CDr \)	13.4	97.2	59.9	48.2	24.2	32.2	96.0	40.6	42.9
Proposed approach
\( \mathcal{M}^2 Trans w/ NLL+BS \)	12.3	87.3	62.4	47.7	46.6	47.2	95.9	41.5	44.1
\( \mathcal{M}^2 Trans w/ NLL+BS+fc_E \)	12.0	99.6	62.6	44.0	53.5	48.3	95.5	44.4	46.8
\( \mathcal{M}^2 Trans w/ NLL+BS+fc_{EN} \)	13.1	103.4	61.0	48.7	46.9	47.8	96.0	43.6	47.1
Table 2: Results of the baselines and our  $\mathcal{M}^2$  Trans model trained with different joint losses. For the metrics, BL4, CDr, and BS represent BLEU4, CIDEr-D, and the  $\mathrm{F_1}$  score of BERTScore; P, R,  $\mathrm{F_1}$  and acc. represent the precision, recall,  $\mathrm{F_1}$ , and accuracy scores output by the clinical CheXbert labeler, respectively. For the rewards,  $\mathrm{fc_E}$  and  $\mathrm{fc_{EN}}$  represent factENT and factENTNLI, respectively.
which is a reward based on the clinical metrics.
R2Gen The model of Chen et al. (2020) with a CNN encoder and a memory-driven Transformer optimized with NLL loss. We used the publicly available official code and its checkpoint as its implementation.
For reproducibility, we include model configurations and training details in Appendix B.
5.1 Evaluation with NLG Metrics and Clinical Metrics
Table 2 shows the results of the baselines and  $\mathcal{M}^2$  Trans optimized with the five different joint losses. We find that the best result for a metric or a reward is achieved when that metric or reward is used directly in the optimization objective. Notably, for the proposed factual rewards, the increases of  $+3.6\mathrm{fact}_{\mathrm{ENT}}$  and  $+4.9\mathrm{fact}_{\mathrm{ENTNLI}}$  are observed
on MIMIC-CXR with  $\mathcal{M}^2$  Trans when compared against  $\mathcal{M}^2$  Trans w/ BS. For the clinical metrics, the best recalls and  $\mathrm{F}_1$  scores are obtained with  $\mathcal{M}^2$  Trans using fact $_{\text{ENT}}$  as a reward, achieving a substantial  $+22.1$  increase ( $\Delta + 63.9\%$ ) in  $\mathrm{F}_1$  score against the best baseline R2Gen. We further find that using fact $_{\text{ENTNLI}}$  as a reward leads to higher precision and accuracy compared to fact $_{\text{ENT}}$  with decreases in the recalls. The best precisions and accuracies were obtained in the baseline CNN-RNN $^2$ . This is not surprising since this model directly optimizes the clinical metrics with its Clinically Coherent Reward. However, this model is strongly optimized against precision resulting in the low recalls and  $\mathrm{F}_1$  scores.
The results of  $\mathcal{M}^2$  Trans without the proposed rewards and BERTScore reveal the strength of  $\mathcal{M}^2$  Trans and the inadequacy of NLL loss and CIDEr for factual completeness and consistency.  $\mathcal{M}^2$  Trans w/ NLL shows strong improvements in the clinical metrics against R2Gen. These improvements are a little surprising since both models are Transformer-based models and are optimized with NLL loss. We assume that these improvements are due to architecture differences such as memory matrices in the encoder of  $\mathcal{M}^2$  Trans. The differ
5These MIMIC-CXR scores have some gaps from the previously reported values with some possible reasons. First, TieNet and CNN-RNN² in Liu et al. (2019) are evaluated on a pre-release version of MIMIC-CXR. Second, we used report-level evaluation for all models, but Chen et al. (2020) tested R2Gen using image-level evaluation.
M2Trans w/ BS
Proposed (simple)	R2Gen
(Chen et al., 2020)	No
difference
36.5%	12.0%	51.5%
ence between NLL and NLL+CDr on  $\mathcal{M}^2$  Trans indicates that NLL and CIDEr are unreliable for factual completeness and consistency.
We performed a human evaluation to further confirm whether the generated radiology reports are factually complete and consistent. Following prior studies of radiology report summarization (Zhang et al., 2020b) and image captioning evaluation (Vedantam et al., 2015), we designed a simple human evaluation task. Given a reference report (R) and two candidate model generated reports (C1, C2), two board-certified radiologists decided whether C1 or C2 is more factually similar to R. To consider cases when C1 and C2 are difficult to differentiate, we also prepared "No difference" as an answer. We sampled 100 reports randomly from the test set of MIMIC-CXR for this evaluation. Since this evaluation is (financially) expensive and there has been no human evaluation between the baseline models, we selected R2Gen as the best previous model and  $\mathcal{M}^2$  Trans w/ BS as the most simple proposed model, in order to be able to weakly infer that all of our proposed models are better than all of the baselines. Table 3 shows the result of the evaluation. The majority of the reports were labeled "No difference" but the proposed approach received three times as much preference as the baseline.
There are two main reasons why "No difference" was frequent in human evaluation. First, we found that a substantial portion of the examples were normal studies (no abnormal observations), which leads to generated reports of similar quality from both models. Second, in some reports with multiple abnormal observations, both models made mistakes on a subset of these observations, making it difficult to decide which model output was better.
5.3 Estimating Clinical Accuracy with Factual Rewards
The integrations of factENT and factENTNLI showed improvements in the clinical metrics. We further examined whether these rewards can be
Table 3: The human evaluation result for randomly sampled 100 reports from the test set of MIMIC-CXR by two board-certified radiologists.
Metric	ρ
BLEU4	0.092
CIDEr-D	0.034
BERTScore	0.155
factENT	0.196
factENTNLI	0.255
Table 4: The Spearman correlations  $\rho$  of NLG metrics and factual metrics against clinical accuracy. The strongest correlation among all metrics is shown is bold.
used to estimate the performance of the clinical metrics to see whether the proposed rewards can be used in an evaluation where a strong clinical information extraction system like CheXbert is not available. Table 4 shows Spearman correlations calculated on the generated reports of NLL+BS. factENTNLI shows the strongest correlation with the clinical accuracy which aligns with the optimization where the best accuracy is obtained with NLL+ BS+factENTNLI. This correlation value is slightly lower than a Spearman correlation which Maynez et al. (2020) observed with NLI for the factual data (0.264). The result suggests the effectiveness of using the factual rewards to estimate the factual completeness and consistency of radiology reports, although the correlations are still limited, with some room for improvement.
5.4 Qualitative Analysis of Improved Clinical Completeness and Consistency
The evaluation with the clinically findings metrics showed improved generation performance by integrating BERTScore, fact<sub>ENT</sub>, and fact<sub>ENTNLI</sub>. As a qualitative analysis, we examined some of the generated reports to see the improvements. Example 1 in Figure 3 shows the improved factual completeness and consistency with BERTScore. The atelectasis is correctly generated and left plural effusion is correctly suppressed with NLL+BS. Example 2 in Figure 4 shows the improved factual completeness with fact<sub>ENTNLI</sub>. The edema is correctly generated and atelectasis is correctly suppressed with NLL+BS+fc<sub>EN</sub>. These examples reveal the strength of integrating the three metrics to generate factually complete and consistent reports.
Despite observing large improvements with our model in the clinical finding metrics evaluation, the model is still not complete and some typical factual errors can be found in their generated reports. For example, Example 3 in Figure 4 includes a comparison of an observation against a previous study as
Images	Reference	R2Gen	M2Trans w/ NLL+BS
Example 1	Large right pleural effusion is unchanged in size. There is associated right basilar atelectasis/scarring, also stable. Healed right rib fractures are noted. On the left, there is persistent apical pleural thickening and apical scarring. Linear opacities projecting over the lower lobe are also compatible with scarring, unchanged. There is no left pleural effusion. There is no pneumothorax. Hilar and cardiomeiastinal contours are difficult to assess, but appear unchanged. Vascular stent is seen in the left axillary/subclavian region.	PA and lateral chest views were obtained with patient in upright position. Analysis is performed in direct comparison with the next preceding similar study of DATE. The heart size remains unchanged and is within normal limits. Unchanged appearance of thoracic aorta. The pulmonary vasculature is not congested. Bilateral pleural effusions are again noted and have increased in size on the right than the left. The left-sided pleural effusion has increased in size and is now moderate in size.	As compared to prior chest radiograph from DATE, there has been interval improvement of the right pleural effusion. There is a persistent opacity at the right lung base. There is persistent atelectasis at the right lung base. There is no left pleural effusion. There is no pneumothorax. The cardiomeiastinal and hi-lar contours are unchanged.
Figure 3: An example of radiology reports generated by R2Gen and by the proposed model with the optimization integrating BERTScore. Repeated sentences are removed from the example to improve readability.
Images	Reference	M2Trans w/NLL+BS	M2Trans w/NLL+BS+fCEN
Example 2	Frontal and lateral radiographs of the chest were acquired. There is new mild interstitial pulmonary edema. A small right pleural effusion may be minimally increased. There is also likely a trace left pleural effusion. There is no focal consolidation. The heart size is not significantly changed. There is no pneumothorax. Midline sternotomy wires are noted.	Assessment is limited by patient rotation. The patient is status post median sternotomy and CABG. Heart size is moderately enlarged. The aorta is tortuous and diffusely calcified. There is mild pulmonary vascular congestion. Small bilateral pleural effusions are present. Patchy opacities in the lung bases likely reflect atelectasis. No pneumothorax is identified. There are no acute osseous abnormalities.	The cardiomeiastinal and hilar contours are stable. The aorta is tortuous. The patient is status post median sternotomy. The heart is mildly enlarged. The aorta is tortuous. The lung volumes are lower compared to the prior chest radiograph. Mild pulmonary edema is present. Small bilateral pleural effusions are present. There is no focal consolidation. No pneumothorax is seen. Median sternotomy wires and mediastinal clips are noted.
Example 3	A right-sided hemodialysis catheter terminates at the right atrium. Again seen are reticular interstitial opacities distributed evenly across both lungs, stable over multiple prior radiographs, previously attributed to chronic hypersensitivity pneumonitis on the chest CT from DATE. The cardiac and mediastinal silhouettes are unchanged. The central pulmonary vessels appear more prominent since the DATE study. Superimposed mild edema cannot be excluded. There is no focal consolidation, pleural effusion, or pneumothorax.	Right-sided dual lumen central venous catheter tip terminates in the lower SVC. Heart size remains mildly enlarged. The mediastinal and hilar contours are unchanged. There is no pulmonary edema. Minimal atelectasis is noted in the lung bases without focal consolidation. No pleural effusion or pneumothorax is seen. There are no acute osseous abnormalities.	The cardiomeiastinal and hilar contours are normal. The lung volumes are low. The lung volumes are present. There is mild pulmonary edema. There is no focal consolidation. No pleural effusion or pneumothorax is seen. A right-sided central venous catheter is seen with tip in the right atrium.
Figure 4: Examples of radiology reports generated by the proposed model with the optimization integrating BERTScore and fact<sub>ENTNLI</sub>. Repeated sentences are removed from the examples to improve readability.
"...appear more prominent since ..." in the reference but our model (or any previous models) can not capture this kind of comparison since the model is not designed to take account the past reports of a patient as input. Additionally, in this example, edema is mentioned with uncertainty as "cannot be excluded" in the reference but the generated report with factENTNLI simply indicates it as "There is mild pulmonary edema".
We proposed two new simple rewards and combined them with a semantic equivalence metric to improve image-to-text radiology report generation systems. The two new rewards make use of radiology domain entities extracted with a named entity recognizer and a weakly-supervised NLI to capture the factual completeness and consistency of the generated reports. We further presented a Transformer-based report generation system that
directly optimizes these rewards with self-critical reinforcement learning. On two open datasets, we showed that our system generates reports that are more factually complete and consistent than the baselines and leads to reports with substantially higher scores in clinical metrics. The integration of entities and NLI to improve the factual completeness and consistency of generation is not restricted to the domain of radiology reports, and we predict that a similar approach might similarly improve other data-to-text tasks.
We would like to thank the anonymous reviewers and the members of the Stanford NLP Group for their very helpful comments that substantially improved this paper.
Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 370-379.
William Boag, Tzu-Ming Harry Hsu, Matthew Mcdermott, Gabriela Berner, Emily Alesentzer, and Peter Szolovits. 2020. Baselines for Chest X-Ray Report Generation. In Proceedings of the Machine Learning for Health NeurIPS Workshop, volume 116, pages 126-140.
Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xi-ang Wan. 2020. Generating radiology reports via memory-driven transformer. In Proceedings of The 2020 Conference on Empirical Methods in Natural Language Processing, pages 1439-1449.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-Memory Transformer for Image Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10575-10584.
Dina Demner-Fushman, Samee Antani, Simpsonl Matthew, and George R. Thoma. 2012. Design and development of a multimodal biomedical information retrieval system. Journal of Computing Science and Engineering, 6(2):168-177.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220.
Christiane Fellbaum, editor. 1998. WordNet: A Lexical Database for English. MIT Press.
Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. 2020. Normalized and geometry-aware self-attention network for image captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10324-10333.
Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019. Image captioning: Transforming objects into words. In Advances in Neural Information Processing, volume 32, pages 11137-11147.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2261-2269.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilicus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn L. Ball, Katie S. Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In The Thirty-Third AAAI Conference on Artificial Intelligence, volume 33, pages 590-597.
Baoyu Jing, Zeya Wang, and Eric Xing. 2019. Show, describe and conclude: On exploiting the structure information of chest x-ray reports. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6570-6580.
Baoyu Jing, Pengtao Xie, and Eric Xing. 2018. On the automatic generation of medical imaging reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2577-2586.
Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. 2019. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(317).
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(16035).
Charles E. Kahn, Curtis P. Langlotz, Elizabeth S. Burnside, John A. Carrino, David S. Channin, David M. Hovsepian, and Daniel L. Rubin. 2009. Toward best practices in radiology reporting. *Radiology*, 252(3):852-856.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference for Learning Representations.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9332-9346.