Preference fine-tuning for factuality in chest X-ray interpretation models without human feedback

Authors: Dennis Hein, Zhihong Chen, Sophie Ostmeier, Justin Xu, Maya Varma, Eduardo Pontes Reis, Arne Edward Michalson MD, Christian Bluethgen, Hyun Joo Shin, Curtis Langlotz, Akshay S Chaudhari
Primary area: foundation or frontier models, including LLMs
Keywords: LLMs, VLMs, preference fine-tuning, RLHF, DPO, DAAs, preference alignment, radiology, chest X-rays

Abstract:
Radiologists play a crucial role by translating medical images into actionable reports. However, the field faces staffing shortages and increasing workloads. While automated approaches using vision-language models (VLMs) show promise as assistants, they require exceptionally high accuracy. Most current VLMs in radiology rely solely on supervised fine-tuning (SFT). Meanwhile, in the general domain, additional preference fine-tuning has become standard practice. The challenge in radiology lies in the prohibitive cost of obtaining radiologist feedback. To address this challenge, we propose an automated pipeline for preference feedback, focusing on chest X-ray (CXR) report generation. Our method leverages publicly available datasets containing pairs of images and radiologist-written reference reports with an LLM-as-a-Judge mechanism, eliminating the need for *additional radiologist feedback*. We evaluate and benchmark five direct alignment algorithms. Our results show up to a 57.4\% improvement in average GREEN scores, a LLM-based metric for evaluating CXR reports, compared to the SFT baseline. We study reward overoptimization via length exploitation, with reports lengthening by up to 3.2x. To assess a potential alignment tax, we benchmark on six additional diverse tasks, finding no significant degradations. A reader study involving four board-certified radiologists indicates win rates of up to 0.62 over the SFT baseline, and macro-averaged F1 scores improve by up to 6.7\%, highlighting the utility of our approach.

Decision: reject
Reviewer scores: 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730261967800

thread_type:
official_review

<thread_object>
creator: reviewer_gh2g

<content>
summary:
This is an empirical analysis paper based on the CheXagent VLM model, which compares the performance of different alignment methods in preference fine-tuning for medical report generation, including DPO, KTO, SimPO, IPO and OROP.

strengths:
The article explores the preference fine-tuning pipeline and significantly enhances the GREEN performance through alignment optimization for the VLM based on the GREEN metric, compared to the baseline SFT.

weaknesses:
1. This is an experimental analysis article, but the generalizability of the experimental conclusions is insufficient, as it only evaluates one medical VLM model, leaving unclear whether the conclusions apply to other models as well. 
2. In preference fine-tuning, the quality of the generated candidates seems critical; however, the overall performance of report generation is not good enough, with metrics like precision and recall generally below 0.5. The SFT baseline struggles to produce high-quality candidates, making it difficult to ensure the chosen candidates are reliable, which raises doubts about the alignment's ability to optimize the model in the desired direction.
3. There is a lack of clinical efficacy (CE) metrics that are often used and are very important in report generation. Although preference fine-tuning optimized for the GREEN metric performs much better than the baseline, the GREEN metric itself is not necessarily what is clinically relevant. Clinicians are more concerned about whether a specific disease in the image has been detected, so a comparison of CE metrics for dozens of diseases in MIMIC-CXR is needed, including precision, recall, and F1-score.
4. This paper is vague in the description of some results or conclusions, lacking in-depth analysis. For example, in the experimental results in Table 5, what are the reasons for the error increase in alignment methods in (e) and (f) compared to the SFT baseline? Additionally, the authors state that "verbosity will limit the clinical utility of generated reports," yet, according to the results, SimPO has the smallest verbosity bias but performs the worst among the alignment methods. What is the relationship between verbosity bias and the quality of the generated reports? This is not clearly explained.
5. The alignment methods lead to performance drops in additional diverse tasks, albeit not significantly. This raises questions about the practicality and reliability of performing alignment on medical VLMs. While alignment improves report generation performance, indicating better model understanding of images, why does it not exhibit advantages in other image understanding tasks? What exactly does alignment change in the model? Is it merely a shift to a radiologist-friendly expression? This is not the focus in the clinical setting, and what we care about more is the model’s general ability to understand images.

questions:
Please refer to Sec. weakness for details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730439592208

thread_type:
official_review

<thread_object>
creator: reviewer_wmbo

<content>
summary:
This paper develops a framework for fine-tuning VLMs without requiring clinician annotation. Using various DAA algorithms, the authors claim to have achieved good performance based on automated scoring metrics. I appreciate their comprehensive study in exploring how different DDA algorithms contribute to model performance, as well as their findings that some RL frameworks exploit the automatic grading system by systematically increasing response length. Overall, the authors have done an excellent job explaining their methodology and results, which provide valuable insights for future research. However, I have questions and concerns about whether this framework genuinely enhances the capacity of VLM models.

strengths:
*Innovative Framework*: The framework allows fine-tuning VLMs without clinician annotation, addressing a key barrier in medical AI by reducing dependency on expert-labeled data.

*Comprehensive Algorithm Comparison*: The study thoroughly evaluates different DAA algorithms, providing insights into how each algorithm uniquely enhances model performance.

*Detailed Methodology and Evaluation*: The paper offers a clear, thorough explanation of its methods and results, making it a valuable resource for future research in VLM fine-tuning and performance evaluation.

weaknesses:
1. One major issue is that you used the GREEN score as the judge for your reward model, which introduces bias and makes the comparison less fair. When I evaluate performance without considering the GREEN score, the improvement across other tasks appears somewhat marginal. Additionally, there is evident over-optimization or reward hacking occurring by simply extending response length. I am not convinced this result is sufficiently robust to demonstrate the effectiveness of this method.

2. My primary concern, which raises doubts about this paper, is that DPO and IPO—the two methods with the highest performance in Table 3—are rated the lowest in expert evaluation. It seems insufficient to attribute this discrepancy to verbosity alone negatively impacting human evaluation. There appears to be a more fundamental issue in these responses, leading clinicians to favor responses generated by SFT. For example, the model produces extensive text, some of which is counterfactual. If this is the case, it contradicts the paper’s objective of improving model factuality through RL.

3. Reproducibility. It would be beneficial to make the code public to enhance reproducibility. At a minimum, a clear summary of the DDA algorithms would assist future researchers.

questions:
1. I don't think general metrics like BERTScore, BLEU-4, and ROUGE-L are particularly helpful here, as they do not assess the factual quality of the response. Consider including these in the supplementary figures.

2. I would like to see more examples to understand why clinicians favor the SFT model. I suspect verbosity is a significant issue, negatively impacting response reliability.

3. A straightforward way to mitigate verbosity issues could be to add a rule-based reward term to DPO, IPO, etc., to see if it improves performance in human ratings. If expert ratings are difficult to get, alternatively, using this method to generally reducing verbosity without affecting evaluation scores could also strengthen the findings. This would be a convincing result that could change my assessment of the paper. 

4. You might also want to cite this paper, the first to perform RLHF fine-tuning in VLM models: Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., ... & Darrell, T. (2023). Aligning large multimodal models with factually augmented RLHF. arXiv preprint arXiv:2309.14525.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730548677669

thread_type:
official_review

<thread_object>
creator: reviewer_p4hs

<content>
summary:
The paper introduces a scalable preference alignment technique using an automated “LLM-as-a-Judge” mechanism, reducing the cost and complexity of obtaining preference data without requiring expert radiologist feedback.

strengths:
1.The paper is well-structured overall, with clear and thoughtfully designed figures that effectively illustrate the concepts and results

2.The paper introduces a scalable preference alignment technique using an automated “LLM-as-a-Judge” mechanism, reducing the cost and complexity of obtaining preference data without requiring expert radiologist feedback.

3.The approach leverages Direct Alignment Algorithms (DAAs), showing significant improvements in multiple metrics over SFT baselines, which enhances the accuracy and clinical relevance of the outputs.

4.The model demonstrates consistent performance across various tasks, avoiding the “alignment tax” phenomenon, which further enhances its applicability in high-stakes medical domains.

weaknesses:
Q1. Please discuss specific theoretical limitations of DAAs that may affect applicability to other medical domains or high-stakes fields beyond medicine. 

Q2.  Specific experiments to test robustness to biased/noisy data are necessary, such as evaluating performance across different age groups or testing with artificially injected label noise.

Q3. How to detect and mitigate potential bias accumulation in the LLM-as-judge approach.

Q4. Please justify the choice of baseline and comparison methods, and suggest 1-2 specific additional relevant baselines you should consider including, explaining why they would be particularly relevant comparisons for this work.

Q5. How does this approach scale in terms of cost and efficiency over time and with larger datasets?

Q6. Please discuss how the proposed approach specifically compares to or differs from Token-level Direct Preference Optimization (TDPO) https://doi.org/10.48550/arXiv.2404.11999, and whether incorporating ideas from TDPO could potentially improve their method.

questions:
Q1. Please discuss specific theoretical limitations of DAAs that may affect applicability to other medical domains or high-stakes fields beyond medicine. 

Q2.  Specific experiments to test robustness to biased/noisy data are necessary, such as evaluating performance across different age groups or testing with artificially injected label noise.

Q3. How to detect and mitigate potential bias accumulation in the LLM-as-judge approach.

Q4. Please justify the choice of baseline and comparison methods, and suggest 1-2 specific additional relevant baselines you should consider including, explaining why they would be particularly relevant comparisons for this work.

Q5. How does this approach scale in terms of cost and efficiency over time and with larger datasets?

Q6. Please discuss how the proposed approach specifically compares to or differs from Token-level Direct Preference Optimization (TDPO) https://doi.org/10.48550/arXiv.2404.11999, and whether incorporating ideas from TDPO could potentially improve their method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696787158

thread_type:
official_review

<thread_object>
creator: reviewer_oxgy

<content>
summary:
This paper proposes a scalable, automated preference alignment technique for chest X-ray (CXR) report generation using publicly available datasets and an LLM-as-a-Judge mechanism. The authors evaluate and benchmark five direct alignment algorithms (DAAs) and show that their approach achieves an overall improvement over the SFT baseline. Additionally, they analyze the alignment tax and clinical implications of the proposed methods.

strengths:
1. The article is well-motivated and clearly written, making it easy to follow.
2. The authors systematically examine how DAAs can enhance factual accuracy from three perspectives, considering both improvements and potential degradations introduced by DAAs.

weaknesses:
1. The method for collecting preference data is relatively simplistic. An ablation study on the impact of different metrics used as the Judge for preference data generation would be valuable.
2. The use of LLMs for preference data collection without human feedback has been explored in prior work, such as RLAIF[1]. A comparison or discussion between the proposed method and RLAIF would strengthen this study.
3. As the authors acknowledge, this work focuses on a single model, raising concerns about whether the experimental results adequately support the study's conclusions.

[1] Lee, Harrison, et al. "RLAIF: Scaling reinforcement learning from human feedback with AI feedback." *arXiv preprint arXiv:2309.00267* (2023).

questions:
Overall, this paper is strong, though it may not fully meet the ICLR criteria. I would consider raising my score if the authors:

1. Compare the impact of metrics beyond GREEN in collecting preference data.
2. Provide a comparative analysis between their method and RLAIF for preference data collection.
3. Conduct additional experiments, such as evaluating their method on other vision-language models (VLMs) like LLM-CXR[1].

[1] Suhyeon Lee, et al. "LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation." In *The Twelfth International Conference on Learning Representations*, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732739170659

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their constructive criticism and timely follow-up to our responses throughout the discussion period.

We have now updated the manuscript by:  
- Including clinical efficacy metrics, further emphasizing the clinical utility of our proposed approach
- Updating the introduction and conclusion to clarify a key contribution: the automated pipeline for preference data generation, enabling preference fine-tuning in this setting which otherwise would not be feasible due to the prohibitive cost of obtaining feedback from radiologists
- Clarifying the choices of CheXagent, GREEN, and DAAs, and connecting our approach with previous work using AI feedback, in Section 3
- Updating sections 4.1 and 4.3 to:
  - Make the error subcategories and reader study results more clear
  - Emphasize the relation between quality and verbosity
- Making section 5, now called *Limitations and Discussion*, much more extensive

We hope that the updated manuscript, and our posted comments, effectively address your concerns. If not, please let us know what other aspects we can improve on. We have ongoing work that we will share with you once finished, and we will remain active here throughout the extended discussion period. 

We thank all reviewers again for their insightful comments which have allowed us to greatly improve our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a scalable, automated preference alignment technique for chest X-ray report generation using publicly available datasets and an "LLM-as-a-Judge" mechanism, empirically analyzing and benchmarking five direct alignment algorithms (DAAs) to improve model performance without requiring expert radiologist feedback.

The reviewers found this paper has strengths in its innovative use of a scalable "LLM-as-a-Judge" mechanism to reduce reliance on expert-labeled data and systematic evaluation of Direct Alignment Algorithms (DAAs). The main weaknesses of the paper include limited generalizability, methodological shortcomings, and insufficient evaluation. Multiple reviewers highlighted concerns about the focus on a single model, raising doubts about the general applicability of the findings. The reliance on the GREEN score introduces bias, leading to verbosity and counterfactual information, undermining the goal of improving factual accuracy (Reviewers wmBo, GH2g). The lack of clinical efficacy (CE) metrics, crucial for medical applications, further weakens the evaluation (Reviewer GH2g). Reviewers also noted insufficient comparisons with related works, unclear explanations of key results, and the need for robustness testing against biased or noisy data (Reviewers oXGy, P4Hs). These limitations reduce the impact and reliability of the proposed methods.

During the discussion, the authors responded by adding clinical efficacy metrics, clarifying key contributions and methods, and expanding the discussion and limitations section for improved clarity. Reviewers appreciated the updates, including clinical efficacy metrics and clarifications on methodology, which enhanced the paper’s clarity and addressed some concerns (Reviewers oXGy, P4Hs, wmBo). However, the study was still criticized for limited generalizability due to reliance on a single, underperforming baseline model and the potential bias introduced by the GREEN score (Reviewers oXGy, GH2g, wmBo). Concerns included insufficient validation with alternative metrics, lack of improvement in downstream tasks, and a focus on benchmarking rather than introducing novel methods (Reviewers wmBo, GH2g, P4Hs). Most reviewers maintained their original scores (Reviewers P4Hs, wmBo, GH2g), while Reviewer oXGy slightly raised their score after acknowledging the updates.

This AC found this work to be meaningful as it proposed a method to fine-tune the radiology report generation models towards expert preferences/feedback (via the "LLM-as-a-Judge"), distinguishing itself from the common reliance on word-level cross-entropy loss. This approach is expected to align the trained model more closely with radiologist perception and decision-making. However, as most reviewers pointed out, this AC agrees that the most significant issue with this work is its limited generalizability due to reliance on a low-performance baseline. While the paper showed that the proposed strategies could significantly improve the baseline, the improved performances still fall short of the SOTA results achieved by other radiology report generation models. It is unclear whether the proposed framework will work for more advanced baseline report generation models. This fails to convincingly demonstrate the broader effectiveness and significance of the work.