The Impact of Auxiliary Patient Data on Automated Chest X-Ray Report Generation and How to Incorporate It
Authors: Aaron Nicolson, Shengyao Zhuang, Jason Dowling, Bevan Koopman


Reviewer paper summaries: This paper investigates incorporating diverse patient data sources into multimodal language models for automated chest X-ray (CXR) report generation. The authors utilize the MIMIC-CXR and MIMIC-IV-ED datasets to integrate detailed patient information such as vital signs, medications, clinical history, and emergency department data to enhance the diagnostic accuracy of generated radiology reports. They introduce methods to convert heterogeneous data types into embeddings that prompt a multimodal language model. Through comprehensive experiments, they demonstrate that using a broader set of patient data from CXR exams and ED records significantly improves the clinical accuracy of the generated reports compared to using only the CXR images.
Summary of strengths: 1. The paper tackles an important problem of enhancing automated CXR report generation by leveraging a diverse set of relevant patient data. This has the potential to improve diagnostic accuracy and clinical utility.
2. The authors introduce novel approaches to integrate heterogeneous data types (numerical, categorical, textual, temporal, image) into unified embeddings that can prompt a multimodal language model. These techniques could be useful for other multimodal machine learning applications as well.
3. The experiments are comprehensive, investigating the impact of various patient data sources individually and in combination. This provides insights into which types of data are most beneficial.
4. The empirical results show significant improvements in clinical accuracy metrics when incorporating the additional patient data compared to a baseline of using only CXR images. This demonstrates the value of the proposed approach.
Summary of weaknesses: 1. While the authors' effort to incorporate diverse patient data sources for enhancing automated chest X-ray report generation is commendable, the motivation and practical value of this approach warrant further scrutiny. Radiologists typically generate reports primarily based on their interpretation of the imaging findings, without necessarily having access to the patient's complete medical history or emergency department records. The additional data sources introduced in this study, such as vital signs, medications, and clinical history, may not be directly observable in the chest X-rays themselves. Consequently, the extent to which these supplementary data points contribute to the accuracy and clinical relevance of the generated reports remains unclear. To strengthen the rationale behind this work, the authors should provide more compelling evidence demonstrating how the integration of these non-imaging data sources aligns with the standard practices and information available to radiologists during the report writing process. Further validation of the proposed approach's practical utility in real-world clinical settings would help solidify the motivation for this research direction.
2. Another important consideration is the potential confounding effects introduced by the MIMIC-IV-ED dataset, which contains data from patients admitted to the emergency department due to severe illnesses. In such cases, patients may present with various comorbidities or receive treatments for conditions unrelated to the findings observable on chest X-rays. The authors should address how they account for and mitigate the influence of these potentially interfering factors on the report generation process.
For instance, there may be scenarios where the chest X-ray report indicates no abnormalities, while the patient's other data sources suggest significant anomalies or the presence of medications prescribed for other diagnoses. In such cases, the inclusion of these additional data points could inadvertently introduce noise or misleading information into the generated reports. The authors need to clarify whether they have implemented any strategies to filter out or prioritize information that is directly relevant to the interpretation of chest X-rays, ensuring that the generated reports remain focused and accurate.
Furthermore, the study would benefit from a more detailed analysis of the relationship between the additional patient data and the specific findings mentioned in the radiologist reports. This could involve examining cases where the generated reports diverge from the ground truth and assessing whether the discrepancies can be attributed to the influence of non-imaging data sources. By carefully considering and addressing these potential confounding factors, the authors can strengthen the robustness and reliability of their proposed approach for automated chest X-ray report generation.

2. While the paper uses data from two public datasets, both are from a single institution. It's unclear how well the findings would generalize to data from other hospitals with different patient populations and clinical practices. Testing on another dataset would strengthen the conclusions.
3. The interpretability and explanation of the model's predictions are not explored in depth. For clinical applications, it is important to understand the key factors influencing the outputs. Future work on enhancing the interpretability would make the approach more viable for real-world use.
4. In the Abstract and Introduction, it would be helpful to clarify that the incorporation of additional patient data is compared to a baseline of using only the CXR images (rather than compared to existing CXR report generation models in general).
Comments, suggestions and typos: 1. When the abbreviation "ED" is first used, better to spell out "emergency department".
2. Better to provide a citation for the "attention dilution" concept in Section 5.
3. It is encouraged to discuss potential future work a bit more in the Conclusion, such as testing the approach on other datasets, exploring alternative model architectures, and studying the impact on radiologist workflows.
Overall assessment scores: 2.5