SlowGuess/NAACL_2021_OCR huggingface dataset:
Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P. Langlotz, Dan Jurafsky
Stanford University
{ysmiura, zyh, ebtsai, langlotz, jurafsky}@stanford.edu
Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural language generation metrics such as CIDEr or BLEU, still suffer from incomplete and inconsistent generations. Here we introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate radiology domain entities consistent with the reference, and one that uses natural language inference to encourage these entities to be described in inferentially consistent ways. We combine these with the novel use of an existing semantic equivalence metric (BERTScore). We further propose a report generation system that optimizes these rewards via reinforcement learning. On two open radiology report datasets, our system substantially improved the  $\mathrm{F}_1$  score of a clinical information extraction performance by  $+22.1$  ( $\Delta + 63.9\%$ ). We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines.
An important new application of natural language generation (NLG) is to build assistive systems that take X-ray images of a patient and generate a textual report describing clinical observations in the images (Jing et al., 2018; Li et al., 2018; Liu et al., 2019; Boag et al., 2020; Chen et al., 2020). Figure 1 shows an example of a radiology report generated by such a system. This is a clinically important task, offering the potential to reduce radiologists' repetitive work and generally improve clinical communication (Kahn et al., 2009).
Figure 1: A (partial) example of a report generated from our system (with “...” representing abbreviated text). The system encodes images and generates text from that encoded representation. Underlined words are disease and anatomy entities. The shaded sentences are an example of a contradictory pair.
Automatic radiology report generation systems have achieved promising performance as measured by widely used NLG metrics such as CIDEr (Vedantam et al., 2015) and BLEU (Papineni et al., 2002) on several datasets (Li et al., 2018; Jing et al., 2019; Chen et al., 2020). However, reports that achieve high performance on these NLG metrics are not always factually complete or consistent. In addition to the use of inadequate metrics, the factual incompleteness and inconsistency issue in generated reports is further exacerbated by the inadequate training of these systems. Specifically, the standard teacher-forcing training algorithm (Williams and Zipser, 1989) used by most existing work can lead to a discrepancy between what the model sees during training and test time (Ran
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288-5304 June 6-11, 2021. ©2021 Association for Computational Linguistics
zato et al., 2016), resulting in degenerate outputs with factual hallucinations (Maynez et al., 2020). Liu et al. (2019) and Boag et al. (2020) have shown that reports generated by state-of-the-art systems still have poor quality when evaluated by their clinical metrics as measured with an information extraction system designed for radiology reports. For example, the generated report in Figure 1 is incomplete since it neglects an observation of atelectasis that can be found in the images. It is also inconsistent since it mentions left-sided pleural effusion which is not present in the images. Indeed, we show that existing systems are inadequate in factual completeness and consistency, and that an imaged-to-text radiology report generation system can be substantially improved by replacing widely used NLG metrics with simple alternatives.
We propose two new simple rewards that can encourage the factual completeness and consistency of the generated reports. First, we propose the Exact Entity Match Reward (fact<sub>ENT</sub>) which captures the completeness of a generated report by measuring its coverage of entities in the radiology domain, compared with a reference report. The goal of the reward is to better capture disease and anatomical knowledge that are encoded in the entities. Second, we propose the Entailing Entity Match Reward (fact<sub>ENTNLI</sub>), which extends fact<sub>ENT</sub> with a natural language inference (NLI) model that further considers how inferentially consistent the generated entities are with their descriptions in the reference. We add NLI to control the overestimation of disease when optimizing towards fact<sub>ENT</sub>. We use these two metrics along with an existing semantic equivalence metric, BERTScore (Zhang et al., 2020a), to potentially capture synonyms (e.g., "left and right" effusions are synonymous with "bilateral" effusions) and distant dependencies between diseases (e.g., a negation like "... but underlying consolidation or other pulmonary lesion not excluded") that are present in radiology reports.
Although recent work in summarization, dialogue, and data-to-text generation has tried to address this problem of factual incompleteness and inconsistency by using natural language inference (NLI) (Falke et al., 2019; Welleck et al., 2019), question answering (QA) (Wang et al., 2020a), or content matching constraint (Wang et al., 2020b) approaches, they either show negative results or are not directly applicable to the generation of radiology reports due to a substantial task and do
main difference. To construct the NLI model for fact<sub>ENTNLI</sub>, we present a weakly supervised approach that adapts an existing NLI model to the radiology domain. We further present a report generation model which directly optimizes a Transformer-based architecture with these rewards using reinforcement learning (RL).
We evaluate our proposed report generation model on two publicly available radiology report generation datasets. We find that optimizing the proposed rewards along with BERTScore by RL leads to generated reports that achieve substantially improved performance in the important clinical metrics (Liu et al., 2019; Boag et al., 2020; Chen et al., 2020), demonstrating the higher clinical value of our approach. We make all our code and the expert-labeled test set for evaluating the radiology NLI model publicly available to encourage future research<sup>1</sup>. To summarize, our contributions in this paper are:
1. We propose two simple rewards for imaged-to-text radiology report generation, which focus on capturing the factual completeness and consistency of generated reports, and a weak supervision-based approach for training a radiology-domain NLI model to realize the second reward.
2. We present a new radiology report generation model that directly optimizes these new rewards with RL, showing that previous approaches that optimize traditional NLG metrics are inadequate, and that the proposed approach substantially improves performance on clinical metrics (as much as  $\Delta + 64.2\%$ ) on two publicly available datasets.
2.1 Image-to-Text Radiology Report Generation
Wang et al. (2018) and Jing et al. (2018) first proposed multi-task learning models that jointly generate a report and classify disease labels from a chest X-ray image. Their models were extended to use multiple images (Yuan et al., 2019), to adopt a hybrid retrieval-generation model (Li et al., 2018), or to consider structure information (Jing et al., 2019). More recent work has focused on generating reports that are clinically consistent and accurate. Liu et al. (2019) presented a system that generates accurate reports by fine-tuning it with their Clinically
https://github.com/ysmiura/ifcc
Coherent Reward. Boag et al. (2020) evaluated several baseline generation systems with clinical metrics and found that standard NLG metrics are ill-equipped for this task. Very recently, Chen et al. (2020) proposed an approach to generate radiology reports with a memory-driven Transformer. Our work is most related to Liu et al. (2019); their system, however, is dependent on a rule-based information extraction system specifically created for chest X-ray reports and has limited robustness and generalizability to different domains within radiology. By contrast, we aim to develop methods that improve the factual completeness and consistency of generated reports by harnessing more robust statistical models and are easily generalizable.
2.2 Consistency and Faithfulness in Natural Language Generation
A variety of recent work has focused on consistency and faithfulness in generation. Our work is inspired by Falke et al. (2019), Welleck et al. (2019), and Matsumaru et al. (2020) in using NLI to rerank or filter generations in text summarization, dialogue, and headline generations systems, respectively. Other attempts in this direction include evaluating consistency in generations using QA models (Durmus et al., 2020; Wang et al., 2020a; Maynez et al., 2020), with distantly supervised classifiers (Krysciński et al., 2020), and with task-specific content matching constraints (Wang et al., 2020b). Liu et al. (2019) and Zhang et al. (2020b) studied improving the factual correctness in generating radiology reports with rule-based information extraction systems. Our work mainly differs from theirs in the direct optimization of factual completeness with an entity-based reward and of factual consistency with a statistical NLI-based reward.
2.3 Image Captioning with Transformer
The problem of generating text from image data has been widely studied in the image captioning setting. While early work focused on combining convolutional neural network (CNN) and recurrent neural network (RNN) architectures (Vinyals et al., 2015), more recent work has discovered the effectiveness of using the Transformer architecture (Vaswani et al., 2017). Li et al. (2019) and Pan et al. (2020) introduced an attention process to exploit semantic and visual information into this architecture. Herdade et al. (2019), Cornia et al. (2020), and Guo et al. (2020) extended this architecture to learn geometrical and other relationships between input
Figure 2: An overview of Meshed-Memory Transformer extended to multiple images.
regions. We find Meshed-Memory Transformer (Cornia et al., 2020)  $(\mathcal{M}^2$  Trans) to be more effective in our radiology report generation task than the traditional RNN-based models and Transformer models (an empirical result will be shown in  $\S 4$ ), and therefore use it as our base architecture.
3.1 Image-to-Text Radiology Report Generation with  $\mathcal{M}^2$  Trans
Formally, given  $K$  individual images  $x_{1\dots K}$  of a patient, our task involves generating a sequence of words to form a textual report  $\hat{y}$ , which describes the clinical observations in the images. This task resembles image captioning, except with multiple images as input and longer text sequences as output. We therefore extend a state-of-the-art image captioning model,  $\mathcal{M}^2$  Trans (Cornia et al., 2020), with multi-image input as our base architecture. We first briefly introduce this model and refer interested readers to Cornia et al. (2020).
Figure 2 illustrates an overview of the  $\mathcal{M}^2$  Trans model. Given an image  $x_{k}$ , image regions are first extracted with a CNN as  $\pmb {X} = \mathrm{CNN}(x_k)$ .  $\pmb{X}$  is then encoded with a memory-augmented attention process  $\mathcal{M}_{\mathrm{mem}}(\pmb {X})$  as
$$
\mathcal {M} _ {\mathrm {m e m}} (\boldsymbol {X}) = \operatorname {A t t} \left(\mathrm {W} _ {\mathrm {q}} \boldsymbol {X}, \boldsymbol {K}, \boldsymbol {V}\right) \tag {1}
$$
$$
\operatorname {A t t} (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}) = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} \boldsymbol {K} ^ {T}}{\sqrt {d}}\right) \boldsymbol {V} \tag {2}
$$
$$
\boldsymbol {K} = \left[ W _ {k} \boldsymbol {X}; \boldsymbol {M} _ {k} \right] \tag {3}
$$
$$
\boldsymbol {V} = \left[ W _ {v} \boldsymbol {X}; \boldsymbol {M} _ {v} \right] \tag {4}
$$
where  $W_{q}, W_{k}, W_{v}$  are weights,  $M_{k}, M_{v}$  are memory matrices,  $d$  is a scaling factor, and  $[*, *]$
is the concatenation operation.  $\mathrm{Att}(Q,K,V)$  is an attention process derived from the Transformer architecture (Vaswani et al., 2017) and extended to include memory matrices that can encode a priori knowledge between image regions. In the encoder, this attention process is a self-attention process since all of the query  $Q$ , the key  $K$ , and the value  $V$  depend on  $X$ .  $\mathcal{M}_{\mathrm{mem}}(X)$  is further processed with a feed forward layer, a residual connection, and a layer normalization to output  $\tilde{\boldsymbol{X}}$ . This encoding process can be stacked  $N$  times and is applied to  $K$  images, and  $n$ -th layer output of  $K$  image will be  $\tilde{\boldsymbol{X}}_{n,K}$ .
The meshed decoder first processes an encoded text  $\mathbf{Y}$  with a masked self-attention and further processes it with a feed forward layer, a residual connection, and a layer normalization to output  $\ddot{\mathbf{Y}}$ .  $\ddot{\mathbf{Y}}$  is then passed to a cross attention  $\mathcal{C}(\tilde{\mathbf{X}}_{n,\mathbf{K}},\ddot{\mathbf{Y}})$  and a meshed attention  $\mathcal{M}_{\mathrm{mesh}}(\tilde{\mathbf{X}}_{N,\mathbf{K}},\ddot{\mathbf{Y}})$  as
$$
\mathcal {M} _ {\text {m e s h}} \left(\tilde {\boldsymbol {X}} _ {\boldsymbol {N}, \boldsymbol {K}}, \ddot {\boldsymbol {Y}}\right) = \sum_ {n} \boldsymbol {\alpha} _ {n} \odot \mathcal {C} \left(\tilde {\boldsymbol {X}} _ {n, \boldsymbol {K}}, \ddot {\boldsymbol {Y}}\right) \tag {5}
$$
$$
\mathcal {C} \left(\tilde {\boldsymbol {X}} _ {n, \boldsymbol {K}}, \ddot {\boldsymbol {Y}}\right) = \max  _ {\boldsymbol {K}} \left(\operatorname {A t t} \left(\mathrm {W} _ {\mathrm {q}} \ddot {\boldsymbol {Y}}, \mathrm {W} _ {\mathrm {k}} \tilde {\boldsymbol {X}} _ {\mathrm {n}, \boldsymbol {K}}, \mathrm {W} _ {\mathrm {v}} \tilde {\boldsymbol {X}} _ {\mathrm {n}, \boldsymbol {K}}\right)\right) \tag {6}
$$
$$
\boldsymbol {\alpha} _ {n} = \sigma \left(W _ {n} [ \boldsymbol {Y}; \mathcal {C} (\tilde {\boldsymbol {X}} _ {n, \boldsymbol {K}}, \ddot {\boldsymbol {Y}}) ] + b _ {n}\right) \tag {7}
$$
where  $\odot$  is element-wise multiplication,  $\max_K$  is max-pooling over  $K$  images,  $\sigma$  is sigmoid function,  $W_{n}$  is a weight, and  $b_{n}$  is a bias. The weighted summation in  $\mathcal{M}_{\mathrm{mesh}}(\tilde{\boldsymbol{X}}_{N,\boldsymbol{K}},\ddot{\boldsymbol{Y}})$  exploits both low-level and high-level information from the  $N$  stacked encoder. Differing from the self-attention process in the encoder, the cross attention uses a query that depends on  $\pmb{Y}$  and a key and a value that depend on  $\pmb{X}$ .  $\mathcal{M}_{\mathrm{mesh}}(\tilde{\boldsymbol{X}}_{N,\boldsymbol{K}},\ddot{\boldsymbol{Y}})$  is further processed with a feed forward layer, a residual connection, and a layer normalization to output  $\tilde{\boldsymbol{Y}}$ . As like in the encoder, the decoder can be stacked  $N$  times to output  $\tilde{\boldsymbol{Y}}_N$ .  $\tilde{\boldsymbol{Y}}_N$  is further passed to a feed forward layer to output report  $\hat{y}$ .
3.2 Optimization with Factual Completeness and Consistency
3.2.1 Exact Entity Match Reward (fact<sub>ENT</sub>)
We designed an F-score entity match reward to capture factual completeness. This reward assumes that entities encode disease and anatomical knowledge that relates to factual completeness. A named entity recognizer is applied to  $\hat{y}$  and the corresponding reference report  $y$ . Given entities  $E_{\mathrm{gen}}$  and  $E_{\mathrm{ref}}$  recognized from  $y_{\mathrm{gen}}$  and  $y_{\mathrm{ref}}$  respectively,
precision (pr) and recall (rc) of entity match are calculated as
$$
\Pr_ {\text {E N T}} = \frac {\sum_ {e \in E _ {\text {g e n}}} \delta (e , E _ {\text {r e f}})}{| E _ {\text {g e n}} |} \tag {8}
$$
$$
\mathrm {r c} _ {\text {E N T}} = \frac {\sum_ {e \in E _ {\text {r e f}}} \delta (e , E _ {\text {g e n}})}{| E _ {\text {r e f}} |} \tag {9}
$$
$$
\delta (e, E) = \left\{ \begin{array}{l l} 1, & \text {f o r} e \in E \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {10}
$$
The harmonic mean of precision and recall is taken as  $\mathrm{fact}_{\mathrm{ENT}}$  to reward a balanced match of entities. We used Stanza (Qi et al., 2020) and its clinical models (Zhang et al., 2020c) as a named entity recognizer for radiology reports. For example in the case of Figure 1, the common entities among the reference report and the generated report are pleural and effusion, resulting in  $\mathrm{fact}_{\mathrm{ENT}} = 33.3$ .
3.2.2 Entailing Entity Match Reward (factENTNLI)
We additionally designed an F-score style reward that expands fact $_{ENT}$  with NLI to capture factual consistency. NLI is used to control the overestimation of disease when optimizing towards fact $_{ENT}$ . In fact $_{ENTNLI}$ ,  $\delta$  in Eq. 10 is expanded to
$$
\phi (e, E) = \left\{ \begin{array}{l l} 1, & \text {f o r} e \in E \wedge \mathrm {N L I _ {e}} (\boldsymbol {P}, h) \neq \text {c o n t r a d i c t i o n} \\ 1, & \text {f o r} \mathrm {N L I _ {e}} (\boldsymbol {P}, h) = \text {e n t a i l m e n t} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {11}
$$
$$
\operatorname {N L I} _ {\mathrm {e}} (\boldsymbol {P}, h) = n l i (\hat {p}, h) \text {w h e r e} \hat {p} = \underset {p \in \boldsymbol {P}} {\arg \max } \operatorname {s i m} (h, p) \tag {12}
$$
where  $h$  is a sentence that includes  $e$ ,  $P$  is all sentences in a counter part text (if  $h$  is a sentence in a generated report,  $P$  is all sentences in the corresponding reference report),  $nli(*,*)$  is an NLI function that returns an NLI label which is one of {entailment, neutral, contradiction}, and  $sim(*,*)$  is a text similarity function. We used BERTScore (Zhang et al., 2020a) as  $sim(*,*)$  in the experiments (the detail of BERTScore can be found in Appendix A). The harmonic mean of precision and recall is taken as factENTNLI to encourage a balanced factual consistency between a generated text and the corresponding reference text. For example in the case of Figure 1, the sentence "The left-sided pleural effusion has increased in size and is now moderate in size." will be contradictory to "There is no left pleural effusion." resulting in pleural and effusion being rejected in  $y_{\mathrm{gen}}$ .
3.2.3 Joint Loss for Optimizing Factual Completeness and Consistency
We integrate the proposed factual rewards into self-critical sequence training (Rennie et al., 2017). An RL loss  $\mathcal{L}_{\mathrm{RL}}$  is minimized as the negative expectation of the reward  $r$ . The gradient of the loss is estimated with a single Monte Carlo sample as
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {R L}} (\theta) = - \nabla_ {\theta} \log P _ {\theta} \left(\hat {y} _ {s p} \mid x _ {1 \dots K}\right) \left(r \left(\hat {y} _ {s p}\right) - r \left(\hat {y} _ {g d}\right)\right) \tag {13}
$$
where  $\hat{y}_{sp}$  is a sampled text and  $\hat{y}_{gd}$  is a greedy decoded text. Paulus et al. (2018) and Zhang et al. (2020b) have shown that a generation can be improved by combining multiple losses. We combine a factual metric loss with a language model loss and an NLG loss as
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {\mathrm {N L L}} + \lambda_ {2} \mathcal {L} _ {\mathrm {R L} _ {-} \mathrm {N L G}} + \lambda_ {3} \mathcal {L} _ {\mathrm {R L} _ {-} \mathrm {F A C T}} \tag {14}
$$
where  $\mathcal{L}_{\mathrm{NLL}}$  is a language model loss,  $\mathcal{L}_{\mathrm{RL\_NLG}}$  is the RL loss using an NLG metric (e.g., CIDEr or BERTScore),  $\mathcal{L}_{\mathrm{RL\_FACT}}$  is the RL loss using a factual reward (e.g., factENT or factENTNLI), and  $\lambda_{*}$  are scaling factors to balance the multiple losses.
3.3 A Weakly-Supervised Approach for Radiology NLI
We propose a weakly-supervised approach to construct an NLI model for radiology reports. (There already exists an NLI system for the medical domain, MedNLI (Romanov and Shivade, 2018), but we found that a model trained on MedNLI does not work well on radiology reports.) Given a large scale dataset of radiology reports, a sentence pair is sampled and filtered with weakly-supervised rules. The rules are prepared to extract a randomly sampled sentence pair  $(s_1$  and  $s_2)$  that are in an entailment, neutral, or contradiction relation. We designed the following 6 rules for weak-supervision.
Entailment 1 (E1) (1)  $s_1$  and  $s_2$  are semantically similar and (2) NE of  $s_2$  is a subset or equal to NE of  $s_1$ .
Neutral 1 (N1) (1)  $s_1$  and  $s_2$  are semantically similar and (2) NE of  $s_1$  is a subset of NE of  $s_2$ .
Neutral 2 (N2) (1) NE of  $s_1$  are equal to NE of  $s_2$  and (2)  $s_1$  include an antonym of a word in  $s_2$ .
Neutral 3 (N3) (1) NE types of  $s_1$  are equal to NE types of  $s_2$  and (2) NE of  $s_1$  is different from NE of  $s_2$ . NE types are used in this rule to
Training Data	#samples	Test Accuracy
RadNLI	MedNLI
MedNLI	13k	53.3	80.9
MedNLI + RadNLI	19k	77.8	79.8
Table 1: The accuracies of the NLI model trained with the weakly-supervised approach. RadNLI is the proposed NLI for radiology reports. The values are the average of 5 runs and the bold values are the best results of each test set.
introduce a certain level of similarity between  $s_1$  and  $s_2$ .
Neutral 4 (N4) (1) NE of  $s_1$  are equal to NE of  $s_2$  and (2)  $s_1$  and  $s_2$  include observation keywords.
Contradiction 1 (C1) (1) NE of  $s_1$  is equal or a subset to NE of  $s_2$  and (2)  $s_1$  is a negation of  $s_2$ .
The rules rely on a semantic similarity measure and the overlap of entities to determine the relationship between  $s_1$  and  $s_2$ . In the neutral rules and the contradiction rule, we included similarity measures to avoid extracting easy to distinguish sentence pairs.
We evaluated this NLI by preparing training data, validation data, and test data. For the training data, the training set of MIMIC-CXR (Johnson et al., 2019) is used as the source of sentence pairs. 2k pairs are extracted for E1 and C1, 0.5k pairs are extracted for N1, N2, N3, and N4, resulting in a total of 6k pairs. The training set of MedNLI is also used as additional data. For the validation data and the test data, we sampled 480 sentence pairs from the validation section of MIMIC-CXR and had them annotated by two experts: one medical expert and one NLP expert. Each pair is annotated twice swapping its premise and hypothesis resulting in 960 pairs and are split in half resulting in 480 pair for a validation set and 480 pairs for a test set. The test set of MedNLI is also used as alternative test data.
We used BERT (Devlin et al., 2019) as an NLI model since it performed as a strong baseline in the existing MedNLI system (Ben Abacha et al., 2019), and used Stanza (Qi et al., 2020) and its clinical models (Zhang et al., 2020c) as a named entity recognizer. Table 1 shows the result of the model trained with and without the weakly-supervised data. The accuracy of NLI on radiology data increased substantially by  $+24.5\%$  with the addition
of the radiology NLI training set. (See Appendix A for the detail of the rules, the datasets, and the model configuration.)