Retrieval Information Injection for Enhanced Medical Report Generation

Authors: Jinghao Liu, Cong Lei, Xiaojun Chang, Linchao Zhu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Medical Report Generation, Contrastive Decoding, Training-free

Abstract:
Automatically generating medical reports is an effective solution to the diagnostic bottleneck caused by physician shortage. Existing methods have demonstrated exemplary performance in generating high-textual-quality reports. Due to the high similarity among medical images as well as the structural and content homogeneity of medical reports, these methods often make it difficult to fully capture the semantic information in medical images. To address this issue, we propose a training-free Retrieval Information injectioN (RIN) method by simulating the process of Multidisciplinary Consultation. The essence of this method lies in fully utilizing similar reports of target images to enhance the performance of pre-trained medical report generation models. Specifically, we first retrieve images most similar to the target image from a pre-constructed image feature database. Then, the reports corresponding to these images are inputted into a report generator of the pre-trained model, obtaining the distributions of retrieved reports. RIN generates final reports by integrating prediction distributions of the pre-trained model and the average distributions of retrieved reports, thereby enhancing the accuracy and reliability of the generated report. Comprehensive experimental results demonstrate that RIN significantly enhances clinical efficacy in chest X-rays report generation task. Compared to the current state-of-the-art methods, it achieves competitive results.

Decision: reject
Reviewer scores: 5, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730556828797

thread_type:
official_review

<thread_object>
creator: reviewer_glqw

<content>
summary:
This paper presents a novel method called Retrieval Information injection (RIN) aimed at enhancing the accuracy and efficacy of automatically generated medical reports. RIN involves retrieving similar images from a pre-constructed database and integrating corresponding report information with the predictions of a pre-trained model. By simulating the process of Multidisciplinary Consultation, the proposed method explores the issues related to cross-modal consistency between medical images and reports, showing competitive clinical efficacy and textual quality (evaluated via CE metrics and NLG metrics).

strengths:
The paper is well-written and insightful. Extensive results demonstrate that RIN improves clinical efficacy in generating reports for chest X-rays, which provides a practical solution to the diagnostic bottleneck caused by physician shortages.

weaknesses:
Despite the paper's clarity, several imprecise arguments and overstatements necessitate revision and clarification:
* The effectiveness of the retrieval process depends heavily on the quality and diversity of the pre-constructed image feature database, which might limit the method's applicability in poorly-documented medical fields.
* The implementation of a report filter in the workflow does not significantly improve the situation where false positive information still exists.
* The paper does not extensively explore the impact of errors in the retrieval process itself as a comparison.

questions:
* Has the method been tested on modalities other than chest X-rays, such as MRIs or CT scans, to assess its adaptability and effectiveness?
* Since medical images are highly similar as mentioned in the paper, is it possible for the workflow to retrieve images that are similar but have distinct symptoms, leading to inaccurate diagnosis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730593345279

thread_type:
official_review

<thread_object>
creator: reviewer_khty

<content>
summary:
The paper presents a retrieval-augmented style of generation (RAG) of radiology reports for chest X-rays. Specifically, radiology reports from similar chest X-rays are used to refine the output of a report generative model using contrastive decoding RAG approaches. A selection is then made between the refined textual output,and  the original predicted report without RAG from image directly by comparing to the retrieved reports from similar chest X-rays. 

The base report generator used is CV2DistilGPT2, the chest x-image image similarity is computed using Bio-VLP encoding, and contrastive decoding is implemented using greedy beam search of width 4. 

Overall, the paper offers limited novelty with a rehash of many of the ideas used in other contemporary papers. Added to this, the use of 14 finding label based training limits the practical utility of such methods also seen from the low F-1 scores even for these 14 findings.

strengths:
The basic idea that RAG improves report generation has been shown for the chest X-ray report case. 
Extensive experiments (take up nearly 50% of the paper) have been performed analyzing factors they deemed important to test. Lexical, semantic and clinical efficacy scores were used for evaluation.

weaknesses:
There are multiple methods that have taken the RAG (retrieval augmented generation) approaches to report generation that haven't been mentioned in related work nor compared to them. These include methods those that first classify the findings in chest X-rays and retrieve related reports that match in terms of findings which are more likely to be better in the selection of reports than those based purely on image features[1]-[2]. At least a comparison to these is validated. Similarly, two other papers referenced below use RAG for chest X-ray report generation which will need comparison[2]-[3]. Finally, the differences with other contrastive decoding approached to RAG may have to be discussed [3]-[4]. In general, the relation of retrieval injection to RAG will have to be explained.

[1]https://arxiv.org/abs/2007.13831
[2] https://arxiv.org/abs/2305.03660 - uses RAG
[3] https://arxiv.org/html/2408.01084v1 - Adaptive contrastive decoding in RAG 
[4] https://arxiv.org/html/2406.17519v1 - Entropy based contrastive ensemble

questions:
1. The terminology used to explain Figure 1 is confusing. You mention text decoders and report generators. Are there referring to the same module or two different modules. If different, this is not reflected in Figure 1. 

2. The use of image encoding features to retrieve similar images needs to be evaluated to see the type of reports retrieved. What is the ratio of overlap of findings of such retrieved reports with the ground truth reports associated with these chest X-rays. Since MIMIC dataset is used, all the chest X-ray images (train-test-validate) should have ground truth reports.

3. Line 296 - Average F-1 score should be based on match to ground truth. Is that what is meant in line 296 or F-1 score is computed relative to which report?

4. Steps 1-6 described in Figure 1 are not very clear. Is image information used only in step 3 or also in step 5?

5. Instead of using Bio-VLP for image-to-image matching why not use it directly to retrieve radiology reports as done in earlier papers with CLIP-based retrieval (https://proceedings.mlr.press/v158/endo21a/endo21a.pdf) since Bio-VLP is a multimodal model?

6. The use of the term 'distribution' to refer to the generated output from report generator is confusing. Are multiple reports coming out in one step from the report generator?

7. Were the results from CheXBert freshly generated for the datasets by the authors or a reuse of numbers quoted from previous work since the ChexBert using the Allen NLP has some dependencies on older CUDA libraries.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614361897

thread_type:
official_review

<thread_object>
creator: reviewer_wuqz

<content>
summary:
This work discusses the challenge of automatically generating medical reports to alleviate the diagnostic bottleneck caused by physician shortage. Existing methods excel at producing high-textual-quality reports. However, the high similarity among medical images and the homogeneity of medical reports' structure and content hinder the full capture of semantic information in images. To overcome this, the authors propose a training-free Retrieval Information injection (RIN) method, which enhances pre-trained medical report generation models by utilizing similar reports of target images. This approach aims to improve the accuracy and clinical efficacy of the generated reports without requiring additional training.

strengths:
1. This work complements the results of the expert model by introducing retrieval information as additional knowledge. Unlike previous generation methods that rely only on image or text information, this method incorporates information from retrieved similar reports and injects the retrieved information into a pre-trained medical report generation model by adjusting the decoding process. This way of combining retrieval and generation is relatively novel in the field of medical report generation.

2. The structure of the manuscript is clear, logical and coherent, and the connection between the contents of each part is natural. When introducing the method, the authors elaborate on the key steps of retrieval information acquisition, injection, and adjustment of decoding process, so that the reader can clearly understand the working principle of the method.

weaknesses:
1. In Section REPORTS RETRIEVAL, the authors claim that medical reports are often noisy, which increases the difficulty of processing and understanding model reports, making it more difficult to retrieve useful retrieval reports from medical images. This work infuses the retrieved information into the pre-trained medical report generation model, which means that the quality of report generation is affected by the retrieval effect. If the retrieved information is inaccurate or not highly relevant, then the injected retrieved information may not be effective in improving the quality of report generation, and may even have a negative impact. This work does not further explain the design and effectiveness of the retrieval method. The authors are advised to further validate the effectiveness of the retrieval model.

2. In the ablation study, as shown in Table 4, the model using Pre-trained Model Prediction Distribution, Retrieved-reports Average Distribution, and Report Filter did not achieve the best results in BLEU-2, BLEU-3, and BLEU-4. The authors are advised to further analyze the reasons for the poor performance of the model

3. The authors are advised to supplement the setting details of hyperparameters, as well as a discussion of model effects using different hyperparameters.

questions:
1. Please further explain the differences between the proposed retrieval module and the existing report retrieval methods.
2. Report generation needs to retrieve k highly relevant reports, how to determine the value of k, and what is the specific value of k used in this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730808293052

thread_type:
official_review

<thread_object>
creator: reviewer_fskn

<content>
summary:
In this work, the authors introduce a training-free Retrieval Information injectioN (RIN) method to enhance the accuracy and effectiveness of medical report generation. Inspired by the Multidisciplinary Consultation process, where multiple experts collaboratively diagnose and analyze a patient's condition, the method retrieves similar images from a database and uses their corresponding reports as references. This method proposes an innovative decoding strategy that enhances the quality of the decoded text by injecting retrieved similar reports into the decoding process. This method has been validated through extensive experiments, demonstrating the effectiveness of both the model and the proposed specific strategies.

strengths:
1.The proposed method is simple and straightforward, distinct from traditional contrastive decoding strategies, and its effectiveness has been demonstrated. The fact that it requires no training makes it particularly attractive.
2.The experiments are comprehensive. In addition to the state-of-the-art (SOTA) analysis, the authors conducted extensive analytical experiments, including ablation studies and evaluations of various parameters within the method.

weaknesses:
1.Although the improvement in results is significant, there is a lack of intuitive explanation or insight into the source of this improvement. Additionally, it remains unclear whether this decode strategy can be applied to other report generation methods.
2.If space permits, I suggest moving the details of the INFORMATION INJECTION (currently at the end of the supplementary materials) into the Methods section. Additionally, the current pseudocode is not detailed enough and should be elaborated further.
3.The experimental results in Table 1 do not reach the current state-of-the-art (SOTA) level [1]. The authors could try to combine more advanced methods to verify the stability of the proposed decoding strategy.
4.In Table 3, it appears that the proposed retrieved-reports average distribution may introduce noise during the decoding process (as discussed in section 3.3), also evidenced by comparing it with the results in the full version (the last row), which included the report filter. Although it achieved significant improvement compared to the baseline (the first row), addressing such adverse effects in certain situations could further strengthen this method.
5.Does this decoding strategy heavily rely on retrieval accuracy?


Reference: 
[1] Jin H, Che H, Lin Y, et al. Promptmrg: Diagnosis-driven prompts for medical report generation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(3): 2607-2615.

questions:
refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732874679098

thread_type:
official_comment

<thread_object>
creator: area_chair_lhsg

<content>
title:
Please review the author response

comment:
Dear Reviewers,

If you have not done so, could you review the author response and let them know if you are satisfied with it or if you have any additional questions?

Kind regards,

Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a method to improve medical report generation. The key idea is to retrieve the reports of the images similar to the input image and inject the information on these reports into the generator to enhance the quality of the output report. Experimental study is conducted to demonstrate the effectiveness of the proposed method. Reviewers comment that this work is straightforward and attractive, has comprehensive experiments, and is well structured and easy to follow. At the same time, reviewers raise issues related to the lack of insight on improvement, the performance with respect to the latest methods, the adverse effect of noise, the reliance on retrieval accuracy, the differences from existing methods, the need of more evaluation on retrieved reports, limited novelty, and so on. The authors provide a rebuttal. The rebuttal does address some of the concerns such as the generality issue, the differences from existing report retrieval methods, experimental settings, and the way of using Bio-VLP. However, the answers to some issues are not informative enough and some issues are not well answered. The key unresolved concerns are 1) the effectiveness and robustness of the retrieval process and how it exactly improves the quality of generated report, 2) the adverse effect of noise and the effectiveness of report filtering shall be better analysed, and 3) the performance on NLG metrics is not well improved or even becomes worse, but this is not sufficiently analysed. By reading the submission, the reviews, the rebuttals, and the message sent by the authors, AC can see that this work has its merits, especially in clearly identifying the issues of current report generation methods and motivating this work. Meanwhile, AC also agrees with the reviewers on the raised concerns. Most of the final ratings are on the negative side. Considering all the factors, this work in its current form cannot be recommended for acceptance. It is hoped that the reviews could help to further improve the quality of this work.