Improving Medical Knowledge in the Automated Chest Radiograph Report
Year: 2021

The clinical writing of unstructured reports from chest radiograph imaging is error prone, due to the lack of its standardization and repeated report writing daily, which can prove fatal. A system that generates reports can assist clinicians to reduce errors. Such a system could further be used as a training tool for medical education as well as used in the global setting to promote medical accessibility in low resource areas. Current medical report generating efforts employ the BLEU metric which was shown to score better clinically meaningless, yet grammatical, random reports [1 2] Further, state of the art methods for this task pretrain the visual extractor on Imagenet which has been shown to generalize poorly for medical domain applications  We seek to study the benefit of pretraining on a chest radiograph specific trained visual extractor. We also combine both feature extractors to study how this extra input information to the generating model can improve the semantic medical accuracy of the resulting reports. We test each model by evaluating BLEU(1 4) metrics and F1 score performance on each of 14 possible labels as defined by CheXpert for chest radiographs. We find that while the chest radiograph feature extrator model and the double feature model result in lower BLEU scores, they perform better across specific Fl scores and total FI score. We provide evidence to suggest that the choice of Imagenet, domain specific, or combined feature extractor is dependent specifically on which medical knowledge is most important for the application. This supports the further investigation of using a combined domain specific feature extractor with an Imagenet pretrained feature extractor for medical imaging captioning tasks. 1 Key Information C/ Mentor: Akshay Smit e External Collaborators (if you have any) Edward Vendrow (Not Enrolled in CS224n) C/ Sharing project: Ethan is sharing the project with CS230 2 Introduction and Related Work In the medical domain, a task that appears in almost all specialities is the generation of reports from medical imaging. Whether this imaging is simple 2D chest radiographs or 3D time series of functional brain activity mappings, experienced clinicians generating many such reports daily are error prone. In a medical setting, such errors could prove fatal. Advances in deep learning based image captioning allow for the potential automation of such clinical tasks. The captioning and report generation for chest radiographs is an emerging area of research with the release in 2019 of MIMIC CXR, a dataset composed of over 220,000 studies with chest radiographs Stanford CS224N Natural Language Processing with Deep Learning and their associated clinical report  State of the art models for such report generations have used transformers, transformer coupled with relational memory, LSTM with reinforcement learning, and retrieval techniques [5 7] Current approaches use pretraining on Imagenet for the encoding of the chest radiograph as input to the generation model. However, recent work has demonstrated that Imagenet pretraining does not transfer well to medical domain tasks  For high parameter models in the medical domain, Imagenet pretraining does not provide a large boost, and more parameter efficient models for chest radiograph specific tasks can be constructed without Imagenet transfer learning  This gap in Imagenet feature learnings and medical domain features was shown to be significantly important in later task performance  In recent work, using radiographs to learn from labelled manual annotations, thereby replacing Imagenet, resulted in outperforming Imagenet based state of the art models  CheXpert is a large dataset of chest radiographs matched with labels for 14 medical conditions: Enlarged Cardiomediastinum, Cardiomegaly, Lung Opacity, Lung Lesion, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, Fracture, Support Devices, No Finding  The large competition on CheXpert performance gives the opportunity to use a domain specific feature extractor for chest radiograph input for later NLP tasks. While state of the art methods, using Imagenet, have resulted in strong performance on standard natural language generation metrics such as BLEU score, evaluation is significantly hampered by the domain agnosticism of such metrics. Novel research demonstrated that such metrics assign high scores to grammatically correct, yet, clinically irrelevant models [1, 2] Therefore, we explore evaluation techniques to emphasize medical semantic performance and study how different feature extraction methods impact such medical semantic model performance. The release of MIMIC CXR dataset inspired multiple efforts for chest radiograph image captioning. The field existed prior to the 2019 release; however, it was limited to datasets using only a few thousands matched image to report examples. With the release of MIMIC CXR containing over 200,000, more advanced models making use of novel transformer architectures catalyzed performance in the growing field. The state of the art model, as quantified by BLEU Scores, used a transformer architecture. It modifies the standard transformer to incorporate the concept of relational memory by allowing the model to pay attention to past cycles during a generating cycle  The improved performance of this transformer based generating model inspired our approach to be transformer based, rather than past LSTM and RL models [6,7, 10]