Year: 2023

Abstract:
Clinical practice frequently uses medical imaging for diagnosis and treatment. A significant challenge for automatic radiology report generation is that the radiology reports are long narratives consisting of multiple sentences for both abnormal and normal findings. Therefore, applying conventional image captioning approaches to generate the whole report proves to be insufficient, as these are designed to briefly describe images with short sentences. We propose a template-based approach to generate radiology reports from radiographs. Our approach involves the following: i) using a multilabel image classifier, produce the tags for the input radiograph; ii) using a transformer-based model, generate pathological descriptions (a description of abnormal findings seen on radiographs) from the tags generated in step (i); iii) using a BERT-based multi-label text classifier, find the spans in the normal report template to replace with the generated pathological descriptions; and iv) using a rule-based system, replace the identified span with the generated pathological description. We performed experiments with the two most popular radiology report datasets, IU Chest X-ray and MIMIC-CXR and demonstrated that the BLEU-1, ROUGE-L, METEOR, and CIDEr scores are better than the State-of-the-Art models by 25%, 36%, 44% and 48% respectively, on the IU X-RAY dataset. To the best of our knowledge, this is the first attempt to generate chest X-ray radiology reports by first creating small sentences for abnormal findings and then replacing them in the normal report template.

heading: 1 Introduction
text: Radiology report generation, which aims to automatically generate a free-text description of a clinical radiograph (like a chest X-ray), has become an important and interesting area of research in both clinical medicine and artificial intelligence. Natural Language Processing (NLP) can speed up
report generation and improve healthcare quality and standardization. Thus, recently, several methods have been proposed in this area (Jing et al., 2017; Li et al., 2018; Yuan et al., 2019; Chen et al., 2020; Alfarghaly et al., 2021). Radiology reports are lengthy narratives, which makes report generation difficult. Therefore, applying conventional image captioning approaches to generate the whole report (Vinyals et al., 2015; Anderson et al., 2018) proves to be insufficient, as such approaches are designed to briefly describe images with short sentences. Further, even if benchmark datasets are balanced between normal and abnormal studies, multiple organs’ findings are included in a single report, and if at least one organ is abnormal, the report is classified as abnormal, but it still contains more normal sentences than abnormal. As a result, existing text generation models may be overly focused on widely reported normal findings. Hence, we propose an approach to generating radiology reports by generating pathological descriptions, i.e., descriptions of abnormal findings, and then replacing corresponding normal descriptions from the normal report template with a generated pathological description. Experimental results on the IU Chest X-ray (referred to as IU X-RAY) and the MIMIC-CXR benchmark datasets confirm the validity and effectiveness of our approach and demonstrate that our approach achieves better results than the State-of-the-Art methods. Two experts in the field were involved in this work. One is a radiologist with 30 years of experience, and the other is a doctor with an MBBS, MD, and Ph.D. and 2 years of experience in medicine.
Our contributions are:
1. A new approach to generating radiology reports: i) generating the tags for the input radiograph; ii) generating pathological descriptions from the tags generated in step (i); iii) identifying the spans in the normal report template to replace with the generated pathologi-
10731
cal descriptions; and iv) replacing the identified span with the generated pathological description—improves the quality and accuracy of the radiology reports. Compared to the previous State-of-the-Art models, the BLEU-1, ROUGE-L, METEOR, and CIDEr scores of the pathological descriptions generated by our approach are raised by 25%, 36%, 44% and 48% respectively, on the IU X-RAY dataset. 2. A dataset of tags (tags are the disease keywords and radiological concepts associated with X-ray images) and their corresponding pathological descriptions. (Derived from IU X-RAY and MIMIC-CXR datasets containing 3827 and 44578 data points respectively.) 3. A dataset of the pathological descriptions and the corresponding normal sentences from the normal report template to replace with pathological descriptions (6501 data points).

heading: 2 Related Work
text: The topic of automatic report generation was researched by Jing et al. (2018); Zhang et al. (2017); Yuan et al. (2019). Several attempts have been made in the medical field to create medical reports from the corresponding images. Most researchers use multilabel image captioning to produce X-ray reports, and they subsequently use those captions as textual features. The IU X-ray dataset was created by Demner-Fushman et al. (2016) to generate radiology reports automatically. The IU X-RAY dataset’s chest X-ray images were used to generate the first structured report using tags predicted by a CNN-RNN model (Shin et al., 2016). A system for generating natural reports for the Chest-Xray14 dataset, employing private reports, was presented by Wang et al. (2017). This framework used a non-hierarchical CNN-LSTM architecture and focused on semantic and visual aspects. Visual attention given to recurrent decoders and convolutionrecurrent architectures (CNN-RNN) was first introduced by Vinyals et al. (2015) on image captioning.
Radiology report generation has recently shifted to transformer-based models (Vaswani et al., 2017; Devlin et al., 2018). Knowledge-driven Encode, Retrieve, and Paraphrase (KERP) (Li et al., 2019) is used for accurate and reliable medical report generation. Yuan et al. (2019) suggests pretraining the encoder with several chest X-ray images to properly recognise 14 typical radiographic observations. According to Chen et al. (2020) proposals,
radiology reports can be generated using a memorydriven transformer, while Pino et al. (2021) suggests a template-based X-ray report generation approach. Pino et al. (2021) clinically defines the templates for each abnormality to indicate its presence or absence. If the generated tags indicate any disease, then the system will choose the corresponding abnormal sentence template. This method cannot generate patient-specific data like anatomical location, size, etc. Wang et al. (2021) proposed a template-based multi-attention report generation model (TMRGM) for normal and abnormal studies. This work uses template-based techniques exclusively for normal reports.
The differences between our work and previous work are as follows: i) Instead of generating the whole report at once, we generate smaller sentences for only abnormal findings and then replace them in the normal report template. ii) Unlike other stateof-the-art models, our methodology does not put excessive emphasis on normal sentences. iii) If the report’s findings are normal, we use a standard template for normal reports.

heading: 3 Methodology
text: The approach that we follow to generate the radiology reports from the radiographs is as follows:
• Generate the tags for the input chest radiograph. • Generate the pathological description from generated tags. • Replace appropriate normal sentences (referred to as normal description) from the normal report template with the generated pathological descriptions.
– Identify the span in the normal report template to replace.
– Replace the identified span with the generated pathological description.

heading: 3.1 Model Architecture
text: Figure 1 illustrates the four main components of our model, which are the image tagger, textgenerator (i.e., transformer), span identifier, and replacement module. The overall description of the four parts is provided below.

heading: 3.1.1 Image Tagger
text: Tagging X-ray images with multiple tags is a multilabel image classification problem. We build our multi-label classifier on top of convolutional neural
network (CNN) architectures such as ResNet-50. Our model takes an X-ray image as input, calculates a score for each of the L labels, and then uses a cutoff threshold to decide which labels to keep. Target in each sample presented a binary vector [y1 j , y2 j , ..., yL
j ], j = 1...n, y ∈ Y and indicates whether a label is present (1) or absent (0). We use the sigmoid activation function in the output layer, and the binary cross-entropy loss function is used to fit the model. The loss is calculated as:
Loss = 1
L
n∑
i=1
[yilog(ŷi) + (1− yi)log(1− ŷi)]
(1) where ŷi is the ith predicted value by model, yi is the corresponding target value, and L is the number of scalar values in the model output.

heading: 3.1.2 Tags Embedding
text: Input tag set X is tokenized as {x1, x2, ..., x|s|}. Input token embeddings is given by, ex = {ex1 , ex2 , ..., ex|n|}. For a token xi, its embedding is exi ∈ Rd, where d is the dimension of the token embeddings. The positional encoding is added to the input embeddings before passing it to the transformer layer. The text embeddings are the sum of the token embeddings and the positional embeddings, i.e., exp = ex+ep, where ep is the positional embeddings.

heading: 3.1.3 Encoder-decoder Architecture
text: Generating pathological descriptions from tags can be looked at as a text generation problem. Given the input tag sequence X = (x1, x2, ..., xs), the goal of general text generation model is to generate the pathological description sequence Y =
(y1, y2, ..., yT ), where S and T are the length of the input and output sequence respectively. The text generation model can be defined as follows:
p(Y |X) = T∏
t=1
p(yt|X,Y<t) (2)
Finally, it generates a pathological description with the highest probability. If the image is labeled as normal then the text generation module generates the sentence No acute abnormality found. Table 3 shows the samples from the dataset that include tags and their corresponding pathological descriptions.

heading: 3.1.4 Span Identification and Replacement Module
text: Span Identification: The span identification module identifies the span from the normal report template that would be replaced with a generated pathological description. To create a standard normal report template, we manually curated a set of sentences per abnormality indicating absence of abnormality (that means normal findings), totaling 23 sentences. With the expert’s opinion, we built the template by examining the reports and picking existing sentences or creating new ones. Due to the fixed nature of sentences, we treat each as a separate label. Figure 2 shows an example of the normal report template with a label for each sentence mentioned in brackets. We create a dataset of pathological description sentences, which we extract from the findings and impressions section of the original dataset, and annotate it with corresponding normal sentence labels (i.e., lung1, lung2, etc.). If a pathological description cannot be replaced with any of the normal sentences, then we annotate that sentence with the extra label.
This normal sentence identification problem can be formulated as a multilabel text classification problem. Consider input sentence as series of words: X = {w1, w2, ..., wn} and for span identification, we need to predict the sentence category Y = {lung1, lung2, ...., extra}. We use a BERT-based multilabel text classifier to identify the normal sentences. The last layer uses a sigmoid activation function to generate the probability of a sample belonging to the corresponding classes. The loss function is the same as equation 1.
Replacement: Once we obtain the list of normal sentences to replace with the generated pathological description, we submit the normal sen-
tences, the normal report template, and the generated pathological description to the replacement module. If a sentence is labeled as extra then we do not replace any normal sentences, but instead add pathological description sentences as an extra part in the report. For example, if the generated pathological description sentence is, Multiple surgical clips are noted., then we add this as an extra part in the report without removing any normal sentences. Replacement module finds the exact match of the identified normal span sentences with normal report sentences and replaces it with generated pathological description. If the generated description is No acute abnormality found. then our replacement algorithm returns the standard normal report template as the generated report. If the span identifier gives multiple sentences to replace by a single sentence of pathological description, then the replacement module replaces the first sentence from the span and removes the remaining span sentences from the template. For example, if generated pathological description is Stable cardiomegaly with large hiatal hernia., then we have to replace No evidence of hernia (lung14) and Heart size is within normal limits. (heart1). Here, our replacement algorithm replaces lung14 sentence with a generated pathological description and removes the heart1 from the template.

heading: 4 Experiments
text: In this section we cover the datasets, evaluation metrics and baselines used for the training and evaluation of our model in detail.

heading: 4.1 Datasets
text: We conduct our experiments on two datasets, which are described as follows:
• IU X-RAY (Demner-Fushman et al., 2016) : a public radiography dataset compiled by Indiana University with 7,470 chest X-ray images and 3,955 reports. Each report has three parts: an impression, which is a title or summary of the report; findings, which contain the report in detail; and manual tags. • MIMIC-CXR (Johnson et al., 2019) : the largest publicly available radiology dataset that consists of 473,057 chest X-ray images and 227,943 reports. For the purpose of our experiments we utilized 44578 reports.
Figure 3 shows the data samples from the IUChest X-ray dataset and the MIMIC-CXR dataset.
We use distinct fields from the original datasets to train different modules. In this section, we give details about experimental settings for each module separately.

heading: 4.1.1 Image Tagger Dataset
text: We consider frontal chest-radiographs as input and target as tags for the IU X-RAY dataset and CheXpert labels for MIMIC-CXR dataset. We build a model using Convolutional Neural Networks (CNN) to analyze each image and classify it with one or more of the 189 labels for IU X-Ray dataset and 14 labels for MIMIC-CXR dataset. Figure 4 and figure 5 shows the distribution of important classes in IU X-RAY dataset and MIMIC-CXR datasets, respectively. For the IU X-RAY dataset out of 3827 samples train, validation and test split
is 3000, 327 and 500 respectively. For the MIMICCXR dataset out of 44578 samples train, validation and test split is 40000, 2000 and 2578 respectively.

heading: 4.1.2 Pathological Description Generator Dataset
text: To train a model we derive two separate datasets from the original IU X-RAY and MIMIC-CXR datasets. Two tasks are involved in creating this dataset:
• We constructed one more dataset that includes unique sentences from the IU X-RAY dataset, and we annotated them as normal or abnormal. The binary classifier dataset contains 5000 samples, each labeled as normal or abnormal by domain experts. We then train a BERT-based binary classifier to classify each sentence as normal or abnormal. • We use the trained model to classify each sentence from findings and impressions into normal and abnormal classes. We remove normal sentences for each report and consider only abnormal sentences as our pathological description.
Table 1 shows the examples of findings from the
IU X-RAY dataset and extracted pathological descriptions from it. The train, validation, and test split is the same as the image tagger dataset.

heading: 4.1.3 Normal Span Identifier Dataset
text: We construct a dataset to identify the sentences to replace from the normal report template with the generated pathological description. The constructed dataset contains sentences from the findings and impressions of the IU X-RAY dataset and their corresponding list of normal sentences to replace. Annotation guidelines are provided by the domain expert. Table 2 shows the guidelines provided by experts. Using those guidelines, we annotate the pathological description sentences. Data annotated by us is verified by a domain expert and corrected if necessary. Table 3 shows the samples from the span identification dataset. Out of total
6500 samples, the train, validation, and test splits are 5000, 500, and 1000, respectively. Table 3 shows the samples from the multilabel text classification dataset that we have constructed.

heading: 4.2 Training Details
text: We adopted the ResNet50 pretrained on Imagenet as the image tagger to produce tags for radiographs. We used frontal radiographs for both the IU X-RAY and the MIMIC-dataset. The IU X-RAY dataset contains 189 labels, and the MIMIC-CXR dataset contains 14 labels. According to the number of labels, the last layer of the image tagger module is different for both modules. We resize the images from both datasets to 224 x 224. We trained our model up to 20 epochs for both datasets. DGX A100-SXM-80GB GPU server takes 100 minutes for a single epoch for the MIMIC-CXR dataset and approximately 3 minutes for the IU X-RAY dataset. The transformer in our proposed model was initialized with pretrained T5-large model weights. The
model was trained under cross entropy loss with the ADAM optimizer. We set the learning rate to 1e-4. We decayed such a rate by a factor of 0.8 per epoch for each dataset and set the beam size to 5 to balance the generation’s effectiveness and efficiency. Maximum input and output sequence lengths were set to 100. We train the transformer model up to 20 epochs for IU X-RAY dataset and up to 15 epochs for the MIMIC-CXR dataset. For one epoch it takes approximately 10 minutes and 60 minutes on a single DGX A100-SXM-80GB GPU for the IU X-RAY and MIMIC-CXR datasets, respectively. The last module was a BERT-based multilabel classifier, which identifies the normal span to replace. We used pretrained BERT weights to initialize our model. There are a total of 24 labels, according to the number of nodes in the last layer change. We train all the models on a DGX A100-SXM-80GB GPU server. For all transformer based models we use hugging face transformer libraries.1

heading: 4.3 Evaluation
text: We evaluate the results of different modules separately. For the image tagger and span identifier modules, we compare the accuracy, AUC score, precision, recall, and F1 scores. The first slot in the table 5 shows the results for the image tagger module, and the second slot shows the results for the span identifier module on the IU X-RAY and MIMIC-CXR datasets.
We compare the performance of our model with previous State-of-the-Art image captioning based
1https://huggingface.co/docs/transformers/ index
methods like CNN-RNN (Vinyals et al., 2015), CDGPT2 (Alfarghaly et al., 2021) and Visual Transformer (Chen et al., 2020) and template based methods such as CNN-TRG (Pino et al., 2021) and TMRGM (Wang et al., 2021). To evaluate the generated pathological descriptions, we consider the pathological descriptions that we extract from original reports as ground truth. To evaluate the generated full reports, we generate templated reports by replacing ground truth pathological description in normal report template and consider it as ground truth. The performance of the aforementioned models is evaluated by conventional natural language generation (NLG) metrics and clinical efficacy (CE) metrics. Clinical Efficacy (CE) metrics provides a quantitative assessment of the quality of generated radiology reports. Clinical efficacy (CE) metrics are calculated by comparing the critical radiology terminology extracted from the generated and reference reports. We use MIRQI2 tool to calculate the precision, recall, and F1 scores to evaluate the model performance for these metrics. NLG metrics such as BLEU (Papineni et al.,
2https://github.com/xiaosongwang/MIRQI
2002), ROUGE (Lin, 2004), CIDEr and METEOR (Banerjee and Lavie, 2005), which primarily focus on measuring n-gram similarities. For CE metrics, precision, recall, and F1-score are used to evaluate model performance. We use the CheXpert (Irvin et al., 2019) to label the generated reports and compare the results with ground truths in 14 different categories of thoracic diseases and support devices. Table 4 shows NLG and CE metrics of the generated pathological description as well as full reports by our model and baseline models compared to gold standard reports.

heading: 4.4 Qualitative Evaluation and Error Analysis
text: This section provides a qualitative analysis performed by a domain expert. The domain expert classified the generated reports into three categories: accurate (reports with most of the vital information), missing details (reports with no false information but missing some vital details), and misleading (reports with false information and an overall incorrect diagnosis).
We provide 500 test samples from the test dataset and their corresponding generated reports to the domain expert for qualitative analysis. The model generated 78% accurate reports, 12% reports with missing information, and 10% with misleading predictions. Further, these random samples were classified into normal and abnormal reports. Out of 183 normal reports, the model generated 99.95% correct reports, 0.0% reports with missing details, and 0.05% misleading reports. Out of 317 abnormal reports, the model could produce 64% accurate reports, 22% of them with missing details, and 14.5% with false reports. Table 6 contains the results of the qualitative analysis. Figure 6 shows the case studies of ground truth reports and generated pathological descriptions and full reports by our method for the above mentioned categories.

heading: 5 Summary, Conclusion and Future Work
text: We present a template-based approach for generating X-ray reports from radiographs. Our model generates small sentences exclusively for abnormalities, which are then substituted in the normal report template to produce a high-quality radiology report. We create a replacement dataset that contains pathological descriptions and their corresponding normal sentences from the normal report template. Our experimental results demonstrate that, compared to the State-of-the-Art models, the BLEU-1, ROUGE-L, METEOR, and CIDEr scores of the full reports generated by our approach are raised by 25%, 36%, 44% and 48%, respectively. Also, clinical evaluation metrics show that our method generates more clinically accurate reports than the State-of-the-Art methods. Unlike other State-of-the-Art models, our methodology does not put excessive emphasis on normal sentences. In the future, we plan to apply the proposed method to generate radiology reports for CT, MRI, etc. For our experiments, we have used all samples from the IU X-RAY dataset. But we have used only 44578 reports out of 227827 reports for the MIMIC-CXR dataset. Our immediate plan is to perform experiments on the whole MIMIC-CXR dataset. Take away from our work is that creating smaller sentences of pathological descriptions and replacing them in the normal template produces better quality reports than generating the whole report at once.
Limitations
Data unbalancing is one of the limitations of our work. In the future, we would like to address this problem by data oversampling or undersampling. For our experiments, we have used all samples from the IU X-ray dataset. But from the MIMIC-CXR dataset, we have used only 44578 reports out of 227827 reports. Our results for the MIMIC-CXR dataset might differ when we use the whole dataset. To evaluate the generated pathological descriptions, we consider the pathological descriptions that we extract from original reports as ground truth. To evaluate the generated full reports, we generate templated reports by replacing ground truth pathological description in normal report template and consider it as ground truth. So it considers the abnormalities from the original reports and the normal sentences from the normal report template. Automatic generation of chest X-ray reports will make it easier for radiologists to diagnose and write reports.
Our model achieved comparable performance with State-of-the-Art models on chest X-ray report generation. In realistic scenarios, it is still a long way from being used clinically.
Ethics Statement
The IU Chest X-ray dataset’s authors used appropriate techniques to de-identify the text reports. Data is anonymized; hence, our model will not disclose information about the patient’s identity. The MIMIC-CXR dataset does not contain any information related to the patient’s identity, like name, age, or address. These reports are also anonymized. Data itself does not reveal the patient’s identity; hence, our model also does not reveal the patient’s
identity.

heading: Acknowledgements
text: We thank the radiologist, Dr. Milind Gune, for his helpful suggestions and feedback.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section: Limitations
3 A2. Did you discuss any potential risks of your work? Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstact, 1, 5
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4.1, 4.1.2, 4.1.3
3 B1. Did you cite the creators of artifacts you used? 4.1, 4.1.1, 4.1.2, 4.1.3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 4.1
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4.1
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Provided data is already in anonymized form.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.3, 4.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 4.1.3
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Annotator is researcher and co-author
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 4.1
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section: Ethic statement
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Annotator is researcher and co-author

heading: Future Work
text: In the future, we plan to apply the proposed method to generate radiology reports for CT, MRI, etc. For our experiments, we have used all samples from the IU X-RAY dataset. But we have used only 44578 reports out of 227827 reports for the MIMIC-CXR dataset. Our immediate plan is to perform experiments on the whole MIMIC-CXR dataset. Take away from our work is that creating smaller sentences of pathological descriptions and replacing them in the normal template produces better quality reports than generating the whole report at once. Limitations
Data unbalancing is one of the limitations of our work. In the future, we would like to address this problem by data oversampling or undersampling. For our experiments, we have used all samples from the IU X-ray dataset. But from the MIMIC-CXR dataset, we have used only 44578 reports out of 227827 reports. Our results for the MIMIC-CXR dataset might differ when we use the whole dataset. To evaluate the generated pathological descriptions, we consider the pathological descriptions that we extract from original reports as ground truth. To evaluate the generated full reports, we generate templated reports by replacing ground truth pathological description in normal report template and consider it as ground truth. So it considers the abnormalities from the original reports and the normal sentences from the normal report template. Automatic generation of chest X-ray reports will make it easier for radiologists to diagnose and write reports. Our model achieved comparable performance with State-of-the-Art models on chest X-ray report generation. In realistic scenarios, it is still a long way from being used clinically