An Efficient LLM Alignment Framework for Automated Radiology Impression Generation

Authors: Tianyang Zhong, Yifan Zhou, Hanqi Jiang, Zhengliang Liu, Jingyuan Chen, Yutong Zhang, Yi Pan, huaqin zhao, Zihao Wu, Yaxuan Yang, Tiezheng Zhang, Bolun Sun, Xin Zhang, Bei Jiang, Linglong Kong, Wei Liu, Xiaoming Zhai, Tianming Liu, Tuo Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: Large language mode, Radiology Impression Generation, Alignment framework, Reinforcement learning

Abstract:
Large language models (LLMs) are typically specialized for domain tasks through supervised fine-tuning, which optimizes LLMs for likelihood-based objectives. While supervised fine-tuning enables LLMs to generate text that conforms to the language style of a specific domain, such as radiology, it often falls short in enhancing the model's ability to perform detailed diagnostic reasoning or tailor reports for individual patients. In this paper, we explore the use of reinforcement learning to better align LLMs with the intricate requirements of radiological practice. By framing the report generation process as sequential decision-making stages, we present Radiology-Guided Reinforcement Optimization (RGRO), a tailored policy optimization framework designed specifically for medical language tasks. RGRO moves beyond conventional likelihood-based training by directly optimizing for radiology-specific objectives, including consistency with radiology findings and adherence to established professional guidelines. Our empirical evaluations demonstrate that RGRO significantly enhances the diagnostic precision and clinical utility of radiology reports generated by LLMs, outperforming supervised fine-tuning methods and state-of-the-art models. Furthermore, RGRO enables the seamless integration of expert radiologist feedback and external diagnostic tools, all without the need for large-scale annotated datasets.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730006411534

thread_type:
official_review

<thread_object>
creator: reviewer_95oj

<content>
summary:
The study introduced radiology-guided reinforcement optimization (RGRO), a framework to finetune LLMs and align with human preference for radiology report generation (generating the impression section from the findings section). The framework utilizes the instruction fin-tuning with low-rank adaptation (LoRA), then reinforcement learning with direct preference optimization (DPO) guided by an LLM parser to align the model outputs with human expert preferences. The authors evaluated RGRO on MIMIC and OpenI, and compared with SFT baselines, and also evaluated the model’s zero-shot performance on a private dataset. The authors claimed that RGRO significantly enhances the diagnostic precision and clinical utility of reports generated by LLMs, and outperforms SFT and other SoTA models.

strengths:
* The authors proposed RGRO, adopting LoRA and DPO, and is specifically tailored for radiology report generation using radiology relevant objectives in its customized reward function. This may help align LLMs with domain specific needs beyond stylish alignment.
* With the LLM parser and DPO, RGRO may mitigate the requirement of collecting large annotated datasets.
* Ablation of understanding the SFT and DPO setting is good.

weaknesses:
* No comparison with other SoTA models even though the authors mentioned that RGRO outperforms state-of-the-art models. It would be nice to have the results of comparison listed in the main context.
* More details of the evaluation dataset setup might be needed. The author mentioned the phase 1 experiment they use a radiology dataset to do pretraining, finetuned using MIMIC and OpenI, then evaluated on the MIMIC and OpenI test split. In phase 2 the model was trained in two ways (but it seems the same?) using XiangYa + MIMIC + OpenO datasets, and evaluated on the XiangYa test split. It is better to make the description clearer, e.g. which dataset you used for phase 1 pretraining, the number of examples in each split / each dataset, why you used different data split for training for each stage, what’s the benefit of experimenting two different sets of experiment? This would be helpful to understand the generalizability of the proposed method.
* The ablation of modified DPO versus standard DPO might also be helpful for readers to understand how much the modification can help.
* Details of the LLM parser, such as the instruction prompts used for preference selection, is important for reproducibility.
* The authors used ROUGE and BERTScore for evaluation, which are helpful NLG metrics yet may not fully capture the clinical utility. Using clinically relevant metrics, or having radiologist to do evaluation would be able to strengthen the authors’ claim of better diagnostic precision and clinical utility.  For example, in this paper https://openaccess.thecvf.com//content/CVPR2023/papers/Tanida_Interactive_and_Explainable_Region-Guided_Radiology_Report_Generation_CVPR_2023_paper.pdf table 1 and 2, also https://arxiv.org/pdf/2311.18260 table 1, these are commonly used NLG metrics and also clinical relevant metrics such as RadGraph F1, from different evaluation perspectives.

questions:
* RL is known to have the risk of overoptimization, how do the authors monitor and control this risk? I.e., when to stop the DPO?
* How was the LLM parser’s performance evaluated in terms of agreement with human preferences?
* It would be nice to discuss how “RGRO focuses on enhancing fine-grained reasoning” via the modified DPO process.
* It seems to me that section 3.1 to 3.3 is a general RL introduction, perhaps it’s better to move them into the appendix if you prefer to keep them, and add more other details (e.g., datasets) into the main context. You may also move the LoRA introduction in 4.1 to the appendix.
* Given that the authors used GPT for evaluation, will the evaluation be stable while changing the GPT version? (i.e., always accept / reject the expert annotated good / bad impression)
* For MIMIC, are you using MIMIC-CXR or MIMIC-IV CXR images? 
* Please fix: I suppose that your description of RGRO-70 and RGRO-80 are reversed? (RGRO-70 should be 70%SFT + 20%DPO I believe)
* Some numbers are pretty close, do authors ensure that the results are statistically significant? E.g. did authors use bootstrapping to compute confidence intervals?
* Did authors see any specific cases that RGRO performed very well / very bad?
* Did authors do a hyperparameter search for DPO? E.g. Beta value in DPO reward function.
* How sensitive is RGRO to the choice of hyperparameters (e.g., β in the DPO loss function)? Did you perform a hyperparameter search, and if so, what were the results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660107215

thread_type:
official_review

<thread_object>
creator: reviewer_kkew

<content>
summary:
The manuscript presents a framework called RGRO for optimizing large language models in generating radiology impressions from findings. The framework consists of two phases: instruction fine-tuning with LoRA and reinforcement learning with DPO, guided by feedback from an LLM parser. The authors evaluated their approach on datasets such as MIMIC and OPEN-I, and demonstrated a comparison between RGRO and SFT.

strengths:
1. Introducing a reinforcement learning method into the report generation task is a reasonable approach to addressing the limitations of next-token prediction tasks in large language models (LLMs) for clinical diagnosis generation.
2. Experiments were conducted on different datasets and in a multi-center setting, providing a robust evaluation.

weaknesses:
1. Problem Definition: In radiology, findings and impressions describe the image from both an observational and diagnostic perspective. I strongly question the significance and feasibility of generating impressions solely from text findings without considering the image. For example, opacities in a chest X-ray may lead to different preferred diagnoses depending on the image feature, clinical indication, and patient context. 
2. Framework Confusion: The framework is somewhat unclear. In the PRELIMINARIES and RADIOLOGY-GUIDED REINFORCEMENT OPTIMIZATION sections, PPO and DPO are mentioned alternately, and the specific method used by RGRO is not clearly defined. Furthermore, the PRELIMINARIES section does not even mention the DPO method.
3. Lack of Novelty: The method lacks innovation, as it appears to be a direct application of DPO to a paired dataset.
4. Lack of Implementation Details: There is insufficient detail regarding the dataset and preprocessing steps, the construction and validation of the LLM parser, and the experimental hyperparameters and training methods.
5. Insufficient Experimental Evaluation: The experimental evaluation is lacking, with no comparison against other methods. The design and analysis of the two experimental groups are also flawed.

questions:
1. As mentioned, is generating impressions solely from text-based findings feasible and useful?
2. What specific method is used during the reinforcement learning phase? This should be clarified.
3. If DPO is used in the second phase, were any modifications or optimizations applied? How does RGRO compare to other DPO frameworks?
4. What are the implementation details of the LLM parser? Specifically, which LLM was used, and how were the prompts designed, aligned, and evaluated?
5. In the experiments involving RGRO, what hyperparameters were used and how were they determined? Additionally, how were checkpoints selected, and what criteria were used? These should are be clarified.
6. In section 5.2, the comparison between the SFT series and the RGRO series seems unfair. For example, SFT-80 uses only 80% of the data, while RGRO-80 uses the entire dataset, with 80% for SFT and 20% for DPO.
7. In section 5.2, RGRO-80 appears to outperform RGRO-70 and RGRO-50. However, according to the manuscript, RGRO-80 consists of 80% SFT and 20% DPO. Does this indicate a trend where less DPO leads to better performance?
8. In section 5.3, the two data splits seem to be set up similarly. What are the differences between these settings?


Minor

1. In line 348, RGRO-80: Utilized 70% SFT and 30% DPO—is this a typo?
2. In Figure 2, under Musculoskeletal Finding: "The liver shows changes consistent with partial hepatectomy, with the chemotherapy pump shadow in the abdominal wall unchanged from previous imaging..."—this is not a musculoskeletal finding.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675802668

thread_type:
official_review

<thread_object>
creator: reviewer_mjyg

<content>
summary:
This paper explores enhancing radiology LLMs through Radiology-Guided Reinforcement Optimization (RGRO), a framework that replaces traditional supervised fine-tuning with a reinforcement learning approach for RRG. RGRO treats report generation as a series of decision-making steps, optimizing for radiology-specific objectives such as diagnostic consistency and adherence to clinical guidelines. Experimental results show that RGRO improves report quality over existing methods.

strengths:
The paper is well-written and relatively easy to follow with the figures. Problem was established and context was provided nicely so the need for RGRO is clear.

weaknesses:
Small things:
- "Fin-tuning" spelt wrong in Figure 1, A.1.
- Table 1 should highlight the best performing cells to make it easier to read

Evaluation with other metrics (aside from just ROUGE and BERTScore) would be desirable as they are not the most suitable for clinical text and there are several other established factuality metrics for radiology and also LLM-based metrics. I recommend the authors do experiments with GREEN (https://arxiv.org/html/2405.03595v1), FineRadScore (https://arxiv.org/html/2405.20613v2), and G-Rad (https://arxiv.org/html/2403.08002v2), as these are relevant.

Perhaps some equations of existing methodolgies are unneeded as they don't add to the overall message of the paper. However, it doesn't really harm the paper - if more space is needed for further experiments or validation, I'd remove some general explanations in Preliminaries and focus more on the modified policy for RGRO.

questions:
Reccomend authors do further validation on their results by using factuality metrics or LLM-based metrics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730830140286

thread_type:
official_review

<thread_object>
creator: reviewer_vzfc

<content>
summary:
The paper presents Radiology-Guided Reinforcement Optimization (RGRO), a framework that uses reinforcement learning to align large language models with the specific requirements of radiological practice, optimizing directly for radiology-focused objectives. 
Empirical results show that RGRO enhances diagnostic precision and clinical utility in radiology reports, compared to traditional supervised methods while incorporating expert feedback and diagnostic tools without large annotated datasets.

strengths:
- The paper proposes Radiology-Guided Reinforcement Optimization (RGRO), a tailored policy optimization framework designed specifically for radiology.
- The papers utilizes three datasets in the experimentation. 
- RGRO- 80 outperforms all other configurations in both ROUGE and BERTScore.

weaknesses:
- There are several new methods that could be tested in addition to DPO and PPO in the ablation experiments.
- Few-shot learning experiments could be tested to comparer the performance with the SFT model etc.in addition to the zero-shot methods.
- The results section only compares SFT with the proposed method. Comparing the papers proposed method with the current DPO and PPO methods could be help with comparison.
- Why is the data trained on the Hospital Data and tested on OpenI and MIMIC only, other variations/data could be trained on?
- The diagram has an error 'Fin-tuning'-->'Fine-tuning'.
- The paper could be written and structured better.

questions:
- It is not clear what the 50,70, and 80 refer too? Why does the 70 have additional annotated data.
- Where clinicians involved in expert incorporation and what particular aspects of expert feedback incorporation was added, such as the format of the feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732359706060

thread_type:
official_comment

<thread_object>
creator: area_chair_lyqv

<content>
title:
Please review author response

comment:
Dear reviewer,

Could you review the author response and let them know if you are satisfied with it or if you have any additional questions?

Kind regards,

Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work uses reinforcement learning to improve the alignment of LLM for medical language tasks. In particular, it proposes a policy optimisation framework called radiology-guided reinforcement learning to automatically generate the impression part from the finding part of a radiology report. Experimental study is conducted on several datasets to show the efficacy of the proposed method. Reviewers comment that this work is well written, presents the motivation clearly, is a reasonable approach, and does a good ablation study. At the same time, reviewers raise issues related to the limited experimental comparison, the need of using more suitable metrics, the soundness of generating impressions from findings, the lack of novelty, and the need of more implementation details, etc. The authors provide a rebuttal. However, the rebuttal is not convincing enough and does not fully address the raised issues, which is also reflected in the reviewers’ comments after reading the rebuttal. All the final ratings are on the negative side. By checking the submission, the reviews, and the rebuttals, AC agrees with the reviewers on the raised concerns. Therefore, this work in its current form cannot be recommended for acceptance.