CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
Authors: Dennis Hein, Zhihong Chen, Sophie Ostmeier, Justin Xu, Maya Varma, Eduardo Pontes Reis, Arne Edward Michalson MD, Christian Bluethgen, Hyun Joo Shin, Curtis Langlotz, Akshay S Chaudhari
Accepted at: acl25

CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback

Dennis Hein1, Zhihong Chen1, Sophie Ostmeier1, Justin Xu1,2, Maya Varma1, Eduardo Pontes Reis1, Arne Edward Michalson1, Christian Bluethgen1,3, Hyun Joo Shin4, Curtis Langlotz1, Akshay S Chaudhari1
1Stanford University, USA 2University of Oxford, UK
3University Hospital Zurich, University of Zurich, Switzerland
4Yongin Severance Hospital, Yonsei University, South Korea
heind@stanford.edu

Abstract

Radiologists play a crucial role in translating medical images into actionable reports. However, the field faces staffing shortages and increasing workloads. While automated approaches using vision-language models (VLMs) show promise as assistants, they require exceptionally high accuracy. Most current VLMs in radiology rely solely on supervised fine-tuning. Meanwhile, additional preference fine-tuning in the post-training pipeline has become standard practice in the general domain. The challenge in radiology lies in the prohibitive cost of obtaining radiologist feedback at scale. To address this challenge, we propose an automated pipeline for preference feedback, focusing on chest X-ray radiology report generation (RRG). Specifically, our method leverages publicly available datasets containing pairs of images and radiologist-written reference reports with reference-based metrics, or Judges, eliminating the need for additional radiologist feedback. We investigate reward overoptimization via length exploitation in this setting and introduce a length-controlled version of the GREEN score. Our best-performing setup achieves state-of-the-art CheXbert scores on the MIMIC-CXR dataset for the RRG task while on average maintaining robust performance across six additional image perception and reasoning tasks.

1 Introduction

X-rays are one of the most frequently collected imaging studies in clinical practice, with the advantages of wide availability, cost-effectiveness, and low radiation dose. Chest X-rays (CXR) are used for diverse purposes, with approximately 1.4 billion diagnostic X-ray examinations collected per year in the world (PAHO, 2012; Organization et al., 2016; Cid et al., 2024). The amount and significance of CXRs can pose a burden for radiologists and a potential negative impact for patients without timely interpretation, especially for those containing critical lesions (Ruuutainen et al., 2013; Hanna et al., 2017; Bruls and Kwee, 2020; Bhargavan et al., 2002; Lyon et al., 2015; Rimmer, 2017).

Recent strides in generative vision-language models (VLMs) hold promising implications for this high-stakes and low-data field (Liu et al., 2024a; Radford et al., 2021). Typically pre-trained using image-text contrastive learning and subsequently fine-tuned, recent VLMs have started to demonstrate promising performance in CXR interpretation and radiology report generation (RRG) (Chen et al., 2024a; Bannur et al., 2024). In high-stakes fields like radiology, where accurate medical descriptions directly influence disease diagnosis and treatment decisions, the generated outputs must maintain high factual accuracy to ensure patient safety.

However, recent studies have shown that supervised fine-tuning (SFT) might be insufficient in the post-training process. For example, Hong et al. (2024) illustrate the limitation of SFT by training on a preference dataset, containing “good” and “bad” completions. By tracking the log probabilities of each during the course of training, they show that the log probabilities of the bad completions inadvertently increase alongside the good completions. Preference fine-tuning methods, such as reinforcement learning from human feedback (RLHF) (Ziegler et al., 2020; Stiennon et al., 2020; Ouyang et al., 2022), using Proximal Policy Optimization (PPO) (Schulman et al., 2017) or REINFORCE (Williams, 1992), and direct alignment algorithms (DAAs), such as Direct Preference Optimization (DPO) (Rafailov et al., 2023), effectively alleviate this problem by employing a negative gradient to lower probabilities of “bad” completions (Tajwar et al., 2024). In fact, most recent large language models (LLMs) (Ouyang et al., 2022; Bai et al., 2022a; Touvron et al., 2023; Jiang et al., 2024; Team et al., 2024) include some form of pref-
Figure 1: Overview of the CheXalign pipeline. For a given dataset containing CXRs and radiologist written references (e.g., MIMIC-CXR), we obtain preference pairs using some automated reference-based metric (e.g., GREEN or the BERTScore), and then optimize the policy using light-weight DAAs (e.g., DPO or KTO).

reference fine-tuning in their post-training pipeline. Yet, this approach has not yet been investigated within the medical vision-language domain.

Yet, this approach has been only sparsely investigated within the medical vision-language domain (e.g., Xiao et al. (2024); Zhu et al. (2025)), concurrent to our work.

The primary challenge hindering the application of preference fine-tuning in the post-training of VLMs in radiology is the prohibitive cost of obtaining radiologist preferences at scale. To overcome this obstacle, we introduce CheXalign, an automated pipeline for generating preference data for the crucial RRG task. Specifically, we leverage the availability of reference reports written by radiologists in a clinical setting within large, publicly available, datasets such as MIMIC-CXR (Johnson et al., 2019) and CheXpert Plus (Chambon et al., 2024). This allows us to use reference-based, uni-modal, metrics, such as GREEN (Ostmeier et al., 2024), a recent state-of-the-art LLM-based metric for evaluating CXR reports, to annotate generated reports in a factually grounded fashion. An overview of our preference fine-tuning pipeline is available in Fig. 1. Our approach enables us to obtain high-quality preference datasets in a fully automated and scalable manner. Using our proposed method, we systematically study how DAAs can be used to enhance the clinical efficacy of medical VLMs without any additional radiologist feedback. Our contributions are as follows:

1. We introduce an automated pipeline for preference pair generation in RRG models, circumventing the prohibitively expensive task of obtaining preference feedback from radiologists at scale.

2. We systematically evaluate and benchmark the proposed pipeline using different reference-based metrics, DAAs, and RRG models. Our findings indicate that the RRG performance can be improved even when using inexpensive, general domain, natural language generation (NLG) metrics for preference pair generation.

3. Using our proposed pipeline, we obtain new state-of-the-art CheXbert scores on the MIMIC-CXR data for the RRG task.

4. We study reward overoptimization via length exploitation, and introduce the length-controlled GREEN score.

5. We benchmark our models post alignment on set of diverse additional image perception and reasoning tasks to assess whether there is an alignment tax in this setting.

Code for this project is available in the following repository: https://github.com/StanfordMIMI/CheXalign.

2 Related Works

VLMs (Radford et al., 2021; Li et al., 2021, 2022, 2023a; Liu et al., 2024a) are a multi-modal extension to LLMs. In this setting, the prompt \( x \) contains images and/or text. Typical tasks include Vision Question Answering (VQA) and image captioning (e.g., RRG in the field of radiology). There is also a line of works to extend VLMs to the medical domain (Thawkar et al., 2023; Hyland et al., 2023; Chaves et al., 2024; Tu et al., 2024; Banur et al., 2024; Chen et al., 2024a,b; Lee et al., 2024b; Jin et al., 2024) which mainly focus on CXR interpretation and RRG due to the wide availability of public datasets (Johnson et al., 2019; Chambon et al., 2024). However, even with strong LLMs and vision-backbones, VLMs have been observed to “hallucinate” and produce outputs that are not factually grounded in the image (Zhou et al., 2024). Such hallucinations represent a significant
risk in high-stakes healthcare fields such as radiology. Similar to Zhou et al. (2024), we pose the problem of hallucinations as an alignment problem and propose tackling it via preference fine-tuning. As noted above, preference fine-tuning, although now a standard part of the post-training pipeline for general domain LLMs, remains largely unexplored in the context of medical VLMs.

RLHF (Ziegler et al., 2020; Stiennon et al., 2020; Ouyang et al., 2022) is a very effective technique for aligning LLMs and VLMs with human preferences. It was instrumental in the development of frontier models such as ChatGPT and Bard (Lee et al., 2024a). RLHF involves two stages: reward modeling, where a reward model is learned on the preference data, and alignment, where reinforcement learning (RL) algorithms are used to optimized the proxy reward. While extremely effective, RLHF is very computationally heavy and can be finicky for non-experts. Relatively recently, a new class of algorithms called DAAs (Rafailov et al., 2024) have become increasingly popular.1 This class of algorithms re-parameterize the reward model via a change-of-variables using the closed-formed solution to the RLHF objective, effectively bypassing both the reward modeling and RL stages. This yields alignment algorithms that remain performant yet computationally more lightweight and significantly easier to implement. DPO (Rafailov et al., 2023) was the first in this category and remains one of the most popular versions. After the advent of DPO, a large variety of DAAs have been suggested (Azar et al., 2023; Park et al., 2024; Ethayarajh et al., 2024; Hong et al., 2024). A brief introduction to RLHF and DAAs is available in §A. Another dimension for improved efficiency is reinforcement learning from AI feedback (RLAIF), first introduced in Bai et al. (2022b). Replacing humans with LLMs leads to significant reductions in cost, making it much more scalable, while maintaining high quality (Lee et al., 2024a). In this work, we will leverage these advancements for scalable, fully automated, preference data generation and computationally lightweight alignment.

Since the reward model in the RLHF objective is learned, it is an imperfect proxy of the ground truth reward. As this proxy is optimized, ground truth performance might saturate or even deteriorate.2 This reward overoptimization, or hacking,

phenomena was first studied in Gao et al. (2023) for RLHF. Despite not fitting an explicit reward model, similar behavior has been observed empirically for DAAs (Rafailov et al., 2024). In particular, length exploitation, the tendency to learn to produce excessively verbose completions, is one common dimension of reward overoptimization, observed in both RLHF and for DAAs. For instance, Park et al. (2024) showed that DPO amplifies minor verbosity bias embedded in the preference data. In this work, we explore this phenomenon in the context of preference fine-tuning of RRG models.

3 Methodology

3.1 RRG Preference Fine-tuning without Human Feedback

Expert human feedback from radiologists is the gold standard for preference data generation and evaluation for the RRG task. However, scaling is impractical, if not unfeasible, due to the limited availability of radiologists for large-scale annotation tasks. In the general domain, it is common to leverage LLMs for cost effective preference data generation (Bai et al., 2022b; Dubois et al., 2023; Lee et al., 2024a). Zheng et al. (2023) categorized “LLM-as-a-Judge” evaluation methods into pairwise, single answer, and reference-guided grading. Pairwise grading being the most common in the general domain both for preference data generation (Dubois et al., 2023; Lee et al., 2024a) and evaluation (Zheng et al., 2023; Dubois et al., 2024).

These existing methods, however, are tailored for uni-modal, general-domain LLMs and do not directly apply to our multi-modal setting, which involves both visual and textual data. Moreover, factual grounding is essential in RRG to ensure clinical reliability. To overcome these challenges, we propose using reference-based grading, leveraging publicly available datasets that contain paired prompts—including images—and radiologist-written reference reports. This abundance of high-quality references allows us to provide factually grounded annotations without the need for a multi-modal metrics, or “Judge”, setting our approach apart from prior studies of preference alignment of VLMs, such as Sun et al. (2024). A comparison of reference-free and reference-based metrics, or “Judges”, in this setting is available in Fig. 2. For a given Judge, we obtain preference pairs by repeat sampling from the SFT baseline. Canonical alignment algorithms, such as DPO, can

1Used more loosely than in Rafailov et al. (2024).
2As per Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.” (Gao et al., 2023).
then be used to preference fine-tune the model.

3.2 Evaluation
Since we have radiologist-written reference available, it is possible to employ standard, general domain, NLG metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and the BERTScore (Zhang et al., 2019). However, these NLG metrics may not be able to differentiate between subtle nuances that are clinically relevant. Thus, we include two validated, radiology-specific, and clinically relevant, metrics: GREEN (Ostmeier et al., 2024) and CheXbert scores (Smit et al., 2020). GREEN is a state-of-the-art metric for radiology report evaluation, based on a single-answer reference-guided LLM-as-a-Judge mechanism. CheXbert scores are a clinical efficacy metrics based on extracting 14 labels5 using the CheXbert labeler (Smit et al., 2020) from the candidate and reference reports.

Initial experimentation illustrated that reward hacking via length exploitation might be a concern in our setting when using the GREEN score as Judge. To counteract this, we propose a simple heuristic approach to explicitly control for length:

\[
\text{LC-GREEN} := \text{GREEN} / \max(\text{rel\_verbosity}, 1),
\]

where rel\_verbosity denotes the relative verbosity (length in words) of the candidate report compared to the reference report. We call this metric length-controlled GREEN (LC-GREEN).

4 Experimental Details

4.1 SFT Baselines
We adopt CheXagent (Chen et al., 2024a) (8B) as a representative example of a state-of-the-art, open source, English language, VLM for the RRG task. It has been trained in a canonical way by first adapting an LLM, Mistral-7B (Jiang et al., 2023), to medical text by continued pre-training. Second, a vision encoder, EVA-CLIP-g (Sun et al., 2023), was adapted via vision pre-training, using contrastive learning on CXR image-text pairs. Third, the two modalities were merged by training a vision-language bridger, or adapter network, keeping the LLM and vision encoder frozen. Finally, the model was instruction-tuned for a range of tasks, including RRG. Concurrent to this work, CheXagent has been greatly improved upon with CheXagent-2 (Chen et al., 2024b) (3B). CheXagent-2 adopted a fine-tuned SigLIP (Zhai et al., 2023) model4 as the vision encoder and a fine-tuned Phi-2 (Li et al., 2023b) model5 (2.7B) as the language decoder. For the image-text connector, instead of using the attention layer as in CheXagent, it uses LLaVA-style MLP connector (Liu et al., 2024b). CheXagent-2 additionally use indications, which provide clinical context to the patient being imaged, to aid in the RRG task.

4.2 Datasets
We use the MIMIC-CXR (Johnson et al., 2019) dataset for training, validation and testing. The image-report pairs consist of one or two CXRs and the corresponding free-text findings section. CheXagent-2 additionally uses the indications. For CheXagent, we randomly sample 80k examples as our training data. For CheXagent-2, we opted to use the full MIMIC-CXR training set6 (148k examples). To test robustness for the RRG task, we additionally include test data from the CheXpert Plus (Chambon et al., 2024) dataset. Moreover, to evaluate whether there is an alignment tax, we additionally evaluate our aligned models on six additional CXR tasks: view classification, binary image classification, single disease identification, multi disease identification, VQA, and image-text reasoning, using test data from five additional datasets RSNA (Shih et al., 2019), SIIM (American College of Radiology, 2019), OpenI (Denner-Fushman et al., 2016), SLAKE (Liu et al., 2021), and Rad-Restruct (Pellegrini et al., 2023) datasets. All datasets are in English.

4.3 Preference Data
We evaluate two reference-based Judges: GREEN (Ostmeier et al., 2024) and the BERTScore (Zhang et al., 2019). We obtain our preference data as follows: 1) for each example in the training data, we prompt the SFT baseline \( N = 4 \) times; 2) we get the score for each of the generated reports, compared with the corresponding singular reference; 3) we set the chosen and rejected completions as the highest and lowest scores, omitting the observation if all

4https://huggingface.co/StanfordAIMI/XraySigLIP_vit-l-16-siglip-384_webli.
5https://huggingface.co/StanfordAIMI/RadPhi-2.
6As can be seen below, CheXagent-2 is a strong SFT baseline and we hypothesized that a larger preference dataset would be required to obtain significant performance gains.
Figure 2: Illustration of preference pairs generation using a reference-free multi-modal Judge (left) and a reference-based uni-modal Judge (right). Human (radiologist) feedback is multi-modal in nature but prohibitively expensive at scale. On the other hand, using LLM, or metrics-based, Judges is highly scalable. However, a high-quality multi-modal Judge is difficult to obtain. For the RRG task, we propose leveraging large publicly available datasets, containing CXRs and radiologist written reference reports, as this enables scalable, factually grounded, preference data generation using reference-based metrics.

<table>
  <tr>
    <th>Algorithm</th>
    <th>Objective</th>
    <th>Preference pairs</th>
    <th>Reference</th>
    <th>Length controlled</th>
    <th>Relative wall-clock time</th>
  </tr>
  <tr>
    <td>DPO</td>
    <td>\( -\log \sigma \left( \beta \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} - \beta \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} \right) \)</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>LC-DPO</td>
    <td>\( -\log \sigma \left( \beta \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} - \beta \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} + \alpha \left| |y_{a}| - |y_{r}| \right| \right), \alpha > 0 \)</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>IPO</td>
    <td>\( \left( \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} - \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} - \frac{1}{2} \right) \)</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>KTO</td>
    <td>\( -\lambda_{1} \sigma \left( \beta \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} - z_{a}d \right) + \lambda_{2} \sigma \left( z_{a}d - \beta \log \frac{\pi_{a}(y|x)}{\pi_{r}(y|x)} \right) \),<br>where \( z_{a}d = \mathbb{E}_{(y|x)\sim D}[D_{KL}\left( \pi_{a}(y|x)||\pi_{r}(y|x) \right)] \)</td>
    <td>×</td>
    <td>✓</td>
    <td>×</td>
    <td>2.2</td>
  </tr>
  <tr>
    <td>ORPO</td>
    <td>\( -\log p_{a}(y|x) - \lambda \log \sigma \left( \log \frac{p_{a}(y|x)}{1-p_{a}(y|x)} - \log \frac{p_{r}(y|x)}{1-p_{r}(y|x)} \right) \),<br>where \( p_{a}(y|x) = \exp \left( \frac{1}{\beta} \log \pi_{a}(y|x) \right) \)</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
    <td>0.7</td>
  </tr>
</table>

Table 1: Overview of the DAAs considered in this paper. Preference pairs indicates whether the method requires paired data of chosen/rejected or only binary feedback indicating whether a completion is desirable/undesirable. Reference indicates whether an additional reference model is loaded during training. Length controlled indicates whether the objective directly controls for the length of the completions in order to mitigate reward hacking via length exploitation. Relative wall-clock time is measured as the total time to train one epoch, relative to DPO.

\( N = 4 \) scores are equivalent. For CheXagent, this rejection rule results in the rejection of 1,246 (1.6%) examples for GREEN and 31 (0.04%) examples for the BERTScore. This is similar for CheXagent-2, with 3744 (2.53%) examples and 147 (0.10%) examples rejected for GREEN and the BERTScore, respectively. Summary statistics for the chosen and rejected subsets are available in Table 8. We also report summary statistics of the length (in words) of the generated reports. Notably, the spread in average length of the chosen and rejected subsets is slightly more pronounced for GREEN than for the BERTScore, 6.9% compared to 5.8% and much more significant overall for CheXagent-2 with a difference of 19.3% and 17.1%, respectively. Examples from the chosen and rejected subsets are available in Fig. 5 and 6.

4.4 Alignment Algorithms

We opt for a representative subset of available, offline, DAAs. DPO is the original DAA and serves as our baseline. In addition to DPO, we consider: 1) Length-controlled DPO (LC-DPO) (Park et al., 2024), as an example of a DAA with explicit length regularization. LC-DPO is an elegant extension of DPO, with an additional hyperparameter \( \alpha \) which controls the strength of an additional length regularization term. Setting \( \alpha = 0 \) reverts the objective to that in DPO. 2) Identity Preference Optimization (IPO) (Azar et al., 2023) as an example of a DAA with generalized preference, relaxing the assumption of the Bradley-Terry model used in DPO. The authors argue that this helps mitigate overfitting issues even when preferences are transitive. Relatively recent work has shown that IPO indeed seems to be less prone to reward overoptimization (Rafailov et al., 2024). 3) Kahneman-Tversky optimization (KTO) (Ethayarajh et al., 2024) as an example of a DAA that does not require preference pairs, but instead only binary feedback on whether a completion is desirable or undesirable. This type of data is much more ubiquitous in practice. In addition, for any given dataset of preference pairs, KTO provides twice the number of examples. 4) Odds-Ratio Preference Optimization (ORPO) (Hong et al., 2024), almost outside of the definition of DAAs, is not based on the RLHF objective but instead appends an additional penalty
Figure 3: Average length against average GREEN and BERTScore for all aligned policies on the MIMIC-CXR validation set using GREEN (left) and the BERTScore (right) as Judges.

directly to the negative log likelihood used in SFT. This adds a “negative gradient”, using the terminology in Tajwar et al. (2024), which will help reduce the log probabilities of rejected completions.

An overview of all DAAs considered in this paper is available in Table 1. More implementation details are available in §B.2.7

5 Results and Analyses

5.1 Length Exploitation

We investigate the issue of length exploitation by plotting the average GREEN and BERTScore against average lengths of the resulting radiology reports using GREEN and the BERTScore as Judges for preference data generation. Results for single runs on the MIMIC-CXR validation set, including all configurations considered, are available in Fig. 3. Save for a IPO run in the lower right corner, there seems to be a positive correlation between average GREEN and average length for CheXagent. In fact, for all DAAs, except for ORPO, there is an indication for a trade-off between added verbosity and GREEN. In addition, such a trade-off is not observed for the BERTScore, shown to the right in Fig. 3. Except for a very tight clustering around 0.87-0.88, there are no clear trends.

To further emphasize this issue, we show the best configurations according to GREEN, and LC-GREEN, in Table 2. For CheXagent, using GREEN to guide the hyperparameter search leads to substantially more verbose reports for DPO, LC-DPO (\( \alpha = 0.001 \)), IPO, and KTO. Qualitative evaluation of the resulting reports indicate that the added verbosity was mostly due to exact, semantically or syntactically, repetitions. This is very likely a manifestation of reward hacking via length exploitation. To mitigate this issue, we use LC-GREEN instead of GREEN to guide the hyperparameter search.\footnote{We opt for this approach, in lieu of obtaining new preference data and re-running the alignment due to computational constraints.}

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">GREEN</th>
    <th colspan="2">LC-GREEN</th>
  </tr>
  <tr>
    <th>Avg. length</th>
    <th>Rel. verbosity</th>
    <th>Avg. length</th>
    <th>Rel. verbosity</th>
  </tr>
  <tr>
    <td>CheXAgent</td>
    <td>55.8</td>
    <td>2.51</td>
    <td>55.8</td>
    <td></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>140.2</td>
    <td>68.7</td>
    <td>123</td>
    <td></td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td>111.5</td>
    <td>53.5</td>
    <td>0.96</td>
    <td></td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>100.2</td>
    <td>58.0</td>
    <td>1.04</td>
    <td></td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td>71.3</td>
    <td>55.9</td>
    <td>1.00</td>
    <td></td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td>63.1</td>
    <td>63.1</td>
    <td>1.13</td>
    <td></td>
  </tr>
  <tr>
    <td>CheXAgent-2</td>
    <td>46.8</td>
    <td></td>
    <td>46.8</td>
    <td></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>60.9</td>
    <td>1.30</td>
    <td>54.7</td>
    <td>1.17</td>
  </tr>
  <tr>
    <td>Reference</td>
    <td>58.4</td>
    <td>1.05</td>
    <td>58.4</td>
    <td>1.05</td>
  </tr>
</table>

Table 2: Average length of reports on the MIMIC-CXR validation set of best performing configurations according to GREEN and GREEN-LC, respectively. Relative verbosity is relative to the SFT baseline.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="3">MIMIC-CXR</th>
    <th colspan="3">CheXpert Plus</th>
  </tr>
  <tr>
    <th>GREEN (↑)</th>
    <th>LC-GREEN (↑)</th>
    <th>BERTScore(↑)</th>
    <th>GREEN (↑)</th>
    <th>LC-GREEN (↑)</th>
    <th>BERTScore(↑)</th>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td></td>
    <td>0.249</td>
    <td>0.218</td>
    <td>0.856</td>
    <td>0.248</td>
    <td>0.202</td>
    <td>0.851</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.325 (30.6)</td>
    <td>0.263 (20.5)</td>
    <td>0.862 (0.65)</td>
    <td>0.322 (29.9)</td>
    <td>0.222 (9.69)</td>
    <td>0.859 (0.91)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.320 (28.7)</td>
    <td>0.288 (32.0)</td>
    <td>0.864 (0.90)</td>
    <td>0.330 (32.8)</td>
    <td><b>0.268 (32.3)</b></td>
    <td>0.861 (1.26)</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>GREEN</td>
    <td>0.326 (30.9)</td>
    <td>0.282 (29.3)</td>
    <td>0.863 (0.84)</td>
    <td>0.334 (34.6)</td>
    <td>0.267 (31.9)</td>
    <td>0.861 (1.15)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td><b>0.328 (31.9)</b></td>
    <td><b>0.293 (34.1)</b></td>
    <td>0.867 (1.27)</td>
    <td><b>0.341 (37.2)</b></td>
    <td>0.266 (31.4)</td>
    <td>0.863 (1.42)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.322 (29.4)</td>
    <td>0.275 (26.2)</td>
    <td>0.862 (0.69)</td>
    <td>0.326 (31.4)</td>
    <td>0.232 (14.6)</td>
    <td>0.856 (0.64)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.285 (14.4)</td>
    <td>0.242 (11.0)</td>
    <td>0.869 (1.56)</td>
    <td>0.278 (12.0)</td>
    <td>0.208 (8.24)</td>
    <td>0.867 (1.89)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.283 (13.5)</td>
    <td>0.259 (18.7)</td>
    <td>0.871 (1.75)</td>
    <td>0.299 (20.3)</td>
    <td>0.248 (22.5)</td>
    <td><b>0.871 (2.34)</b></td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>BERTScore</td>
    <td>0.283 (13.6)</td>
    <td>0.247 (13.1)</td>
    <td>0.870 (1.62)</td>
    <td>0.282 (13.6)</td>
    <td>0.218 (7.88)</td>
    <td>0.866 (1.75)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>0.304 (21.9)</td>
    <td>0.279 (28.2)</td>
    <td><b>0.872 (1.88)</b></td>
    <td>0.308 (24.0)</td>
    <td>0.249 (23.1)</td>
    <td>0.869 (2.11)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.291 (16.9)</td>
    <td>0.265 (21.4)</td>
    <td>0.869 (1.57)</td>
    <td>0.298 (19.9)</td>
    <td>0.240 (18.4)</td>
    <td>0.870 (2.21)</td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td></td>
    <td>0.326</td>
    <td>0.297</td>
    <td>0.888</td>
    <td>0.349</td>
    <td>0.304</td>
    <td>0.892</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>GREEN</td>
    <td><b>0.387 (18.9)</b></td>
    <td><b>0.339 (14.1)</b></td>
    <td>0.891 (0.30)</td>
    <td><b>0.387 (10.9)</b></td>
    <td><b>0.320 (5.34)</b></td>
    <td>0.888 (-0.38)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>BERTScore</td>
    <td>0.352 (8.11)</td>
    <td>0.326 (9.58)</td>
    <td><b>0.896 (0.95)</b></td>
    <td>0.359 (2.88)</td>
    <td>0.310 (1.78)</td>
    <td><b>0.893 (0.15)</b></td>
  </tr>
</table>

Table 3: Results on the MIMIC-CXR and CheXpert Plus test sets (percentage change compared to SFT baseline in brackets). Best results in bold, separate for CheXagent and CheXagent-2.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="2">F1-14</th>
    <th colspan="2">F1-5</th>
    <th rowspan="2">Avg. (↑)</th>
  </tr>
  <tr>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td></td>
    <td>20.4</td>
    <td>35.5</td>
    <td>19.6</td>
    <td>25.8</td>
    <td>25.3</td>
  </tr>
  <tr>
    <td>MAIRA-1</td>
    <td></td>
    <td>38.6</td>
    <td>55.7</td>
    <td>47.7</td>
    <td>56.0</td>
    <td>49.5</td>
  </tr>
  <tr>
    <td>MAIRA-2</td>
    <td></td>
    <td>41.6</td>
    <td>58.1</td>
    <td>50.4</td>
    <td>59.1</td>
    <td>52.3</td>
  </tr>
  <tr>
    <td>Med-PaLM M (12B)</td>
    <td></td>
    <td>37.3</td>
    <td>51.4</td>
    <td>50.6</td>
    <td>56.5</td>
    <td>49.0</td>
  </tr>
  <tr>
    <td>Med-PaLM M (84B)</td>
    <td></td>
    <td>39.8</td>
    <td>53.6</td>
    <td>51.6</td>
    <td>57.9</td>
    <td>50.7</td>
  </tr>
  <tr>
    <td>Med-PaLM M (562B)</td>
    <td></td>
    <td>37.8</td>
    <td>51.6</td>
    <td>49.9</td>
    <td>56.3</td>
    <td>48.9</td>
  </tr>
  <tr>
    <td>LLaVA-Rad</td>
    <td></td>
    <td>39.5</td>
    <td>57.3</td>
    <td>47.7</td>
    <td>57.4</td>
    <td>50.5</td>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td></td>
    <td>38.9</td>
    <td>50.9</td>
    <td>47.6</td>
    <td>54.1</td>
    <td>47.9</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>41.5* (6.87)</td>
    <td>54.1* (6.18)</td>
    <td>51.8* (8.83)</td>
    <td>58.3* (7.78)</td>
    <td>51.4* (7.43)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>37.6 (-3.32)</td>
    <td>52.1 (2.20)</td>
    <td>47.5 (-0.02)</td>
    <td>55.2* (1.92)</td>
    <td>48.1 (0.45)</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>GREEN</td>
    <td>39.0 (0.39)</td>
    <td>52.9* (3.87)</td>
    <td>48.9* (2.88)</td>
    <td>56.5* (4.36)</td>
    <td>49.3 (3.06)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>40.7 (4.81)</td>
    <td>55.0* (8.07)</td>
    <td>51.6* (8.52)</td>
    <td>59.3* (9.54)</td>
    <td>51.7* (7.93)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>41.4* (6.64)</td>
    <td>55.0* (8.03)</td>
    <td>51.5* (8.20)</td>
    <td>58.3* (7.69)</td>
    <td>51.5* (7.69)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>42.5* (9.37)</td>
    <td>56.1* (10.1)</td>
    <td>54.7* (15.0)</td>
    <td>61.6* (13.8)</td>
    <td>53.7* (12.2)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>42.3* (8.75)</td>
    <td>56.3* (10.6)</td>
    <td>52.7* (10.8)</td>
    <td>60.8* (12.4)</td>
    <td>53.0* (10.8)</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>BERTScore</td>
    <td>43.1* (10.8)</td>
    <td>57.0* (11.9)</td>
    <td>53.6* (12.6)</td>
    <td>61.5* (13.5)</td>
    <td>53.8* (12.4)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>44.0* (12.1)</td>
    <td>56.7* (11.4)</td>
    <td>52.5* (10.5)</td>
    <td>60.7* (12.2)</td>
    <td>53.1* (10.9)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>42.4* (9.16)</td>
    <td>56.7* (11.4)</td>
    <td>52.5* (10.5)</td>
    <td>60.7* (12.2)</td>
    <td>53.1* (10.9)</td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td></td>
    <td>44.6</td>
    <td>57.8</td>
    <td>55.5</td>
    <td>62.4</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>GREEN</td>
    <td><b>45.8 (2.6)</b></td>
    <td><b>59.7* (3.3)</b></td>
    <td><b>56.0 (0.8)</b></td>
    <td><b>64.1 (2.7)</b></td>
    <td><b>56.4 (2.4)</b></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>BERTScore</td>
    <td>43.0 (-3.6)</td>
    <td>59.7* (3.3)</td>
    <td>53.5 (-3.6)</td>
    <td>63.8 (2.3)</td>
    <td>55.0 (-0.1)</td>
  </tr>
</table>

Table 4: CheXbert scores on the MIMIC-CXR test set. Percentage change compared to SFT baseline in brackets. Best results in bold. *statistically significantly different from SFT baseline at a 10% level based on confidence intervals in Table 11.

5.2 Judge Optimization Results

Results, using single runs, for GREEN, LC-GREEN, and the BERTScore on the MIMIC-CXR and CheXpert Plus test sets are available in Table 3. Additional results for ROUGE-L and BLEU-4 are available in Table 10. As expected, using GREEN as Judge results in the largest boost in GREEN, and using the BERTScore in the BERTScore. In particular, the top performing configuration on the MIMIC-CXR test set for CheXagent according to GREEN and LC-GREEN is KTO, using GREEN as Judge, boosting these metrics by 31.9% and 34.1% percent, respectively. For CheXagent, the top performing configuration according to the BERTScore is obtained by KTO, using the BERTScore as Judge. We observe similar trends for CheXagent-2 with DPO yielding an increase of 18.9% and 14.1% for GREEN and LC-GREEN, respectively when using GREEN as Judge. We observed overall similar trends for the CheXbert Plus, despite representing two different distributions: MIMIC-CXR was collected in an emergency department and CheXpert Plus was collected from in- and out-patient centers.

5.3 Generalization to CheXbert Scores

Although GREEN is a high quality and clinically relevant metric, the observed performance gains might be inflated due to the fact that we used GREEN as Judge for preference data generation. Thus, we instead turn to the CheXbert scores to be our silver-standard, a low-cost approximation of expert human (radiologists) judgment. CheXbert
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="6">Accuracy (†)</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>View Classification</th>
    <th>Binary Image Classification</th>
    <th>Single Disease Identification</th>
    <th>Multi Disease Identification</th>
    <th>Visual Question Answering</th>
    <th>Image-Text Reasoning</th>
  </tr>
  <tr>
    <td>CheXAgent</td>
    <td></td>
    <td>98.5</td>
    <td>83.8</td>
    <td>62.8</td>
    <td>69.0</td>
    <td>75.5</td>
    <td>66.3</td>
    <td>76.0</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>98.5</td>
    <td>84.0</td>
    <td>63.2</td>
    <td>68.2</td>
    <td>75.5</td>
    <td>66.1</td>
    <td>76.0</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>98.5</td>
    <td>83.7</td>
    <td>62.2</td>
    <td>68.3</td>
    <td>75.5</td>
    <td>65.3</td>
    <td>75.6</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>GREEN</td>
    <td>98.3</td>
    <td>83.8</td>
    <td>62.7</td>
    <td>68.3</td>
    <td>76.8</td>
    <td>65.5</td>
    <td>75.9</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>98.3</td>
    <td>83.8</td>
    <td>63.2</td>
    <td>68.7</td>
    <td>74.6</td>
    <td>65.0</td>
    <td>75.6</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>98.2</td>
    <td>83.5</td>
    <td>63.5</td>
    <td>68.6</td>
    <td>74.3</td>
    <td>64.2</td>
    <td>75.4</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>98.5</td>
    <td>84.3</td>
    <td>62.8</td>
    <td>68.4</td>
    <td>75.1</td>
    <td>66.8</td>
    <td>76.0</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>98.5</td>
    <td>84.1</td>
    <td>63.2</td>
    <td>68.3</td>
    <td>75.1</td>
    <td>65.5</td>
    <td>75.8</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>BERTScore</td>
    <td>98.5</td>
    <td>83.8</td>
    <td>62.9</td>
    <td>68.2</td>
    <td>76.0</td>
    <td>66.3</td>
    <td>76.0</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>98.5</td>
    <td>84.2</td>
    <td>63.2</td>
    <td>68.5</td>
    <td>76.3</td>
    <td>66.3</td>
    <td>76.2</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>98.3</td>
    <td>84.1</td>
    <td>62.7</td>
    <td>68.8</td>
    <td>73.9</td>
    <td>66.3</td>
    <td>75.7</td>
  </tr>
  <tr>
    <td>CheXAgent-2</td>
    <td></td>
    <td>99.0</td>
    <td>83.0</td>
    <td>65.5</td>
    <td>83.9</td>
    <td>83.2</td>
    <td>78.7</td>
    <td>82.2</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>GREEN</td>
    <td>99.2</td>
    <td>84.8</td>
    <td>66.7</td>
    <td>84.4</td>
    <td>80.5</td>
    <td>77.9</td>
    <td>82.2</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>BERTScore</td>
    <td>99.2</td>
    <td>83.0</td>
    <td>65.9</td>
    <td>84.5</td>
    <td>82.3</td>
    <td>78.2</td>
    <td>82.2</td>
  </tr>
</table>

Table 5: Performance on six image perception and reasoning tasks different from RRG on the datasets listed in §4.2.

scores are ubiquitous in this setting and we include results from a representative sample of recent state-of-the-art medical VLMs for the RRG tasks: GPT-4V9, MAIRA-1 (Hyland et al., 2023), MAIRA-2 (Bannur et al., 2024), Med-PaLM (Tu et al., 2024), and LLava-Rad (Chaves et al., 2024). We report macro and micro averaged F1-scores for the full 14 categories as well as the 5 categories subset. In addition, we provide the average across these scores. Results for our method are averages based on 1000 bootstrap samples, with confidence intervals available in Table 11.

For CheXagent, which is comparable to LLava-Rad and MAIRA-1 prior to alignment, our method boosts the average CheXbert scores by up to 7-8% and 14% using GREEN and the BERTScore, respectively. Very interestingly, and unexpectedly, the top performing setup for CheXagent using the BERTScore as Judge is better than that for GREEN as Judge. One possible explanation for this is reward overoptimization, as this seems to be a more prominent issue for GREEN than for the BERTScore. For CheXagent-2, which is the overall state-of-the-art prior to alignment and second only to MAIRA-210 in terms of micro F1-14 scores, we can see that it is possible to improve performance even further using our proposed method. We note, however, that only the micro F1-14 scores are statistically significantly different from CheXbert-2, at a 10% level, for both GREEN and the BERTScore as Judge. Thus, we depict that it is possible to improve upon the already very strong micro averaged F1-scores, even when employing a general domain NLG metric like the BERTScore. Granular results, for each of the 14 categories, are available in Table 12.

5.4 Alignment Tax Analysis

While RLHF is powerful, it has been observed that it might lead to performance degradations or, forgetting (Askell et al., 2021; Ouyang et al., 2022). Ouyang et al. (2022) assessed such an alignment tax by evaluating the aligned policies on several natural language processing (NLP) benchmarks. Inspired by this, we benchmark the SFT baseline and the aligned policies on six diverse image perception and reasoning tasks using datasets listed in §4.2. Although there are some minor variations, on average, the performance matches that of the SFT baselines. Thus, our method substantially improves the quality of generated reports without compromising the quality of other image understanding tasks.

5.5 Qualitative Analysis

To further shed light on policy behavior pre and post alignment, a qualitative study was conducted by selecting three interesting cases from the MIMIC-CXR test set, covering a range of positive and negative findings and the presence and absence of indications. A board-certified radiologist was then asked to color-code candidate reports from the SFT baselines and aligned policies. Results for CheXagent-2 using GREEN as Judge for the first example are available in Fig. 4. For this case, despite being very strong, CheXagent-2 exhibited two errors: 1) it incorrectly reported clear lungs and 2) it failed to detect small bilateral pleural effusions. Notably, the aligned version addressed both of these errors. A similar conclusion is true for

9https://openai.com/index/gpt-4v-system-card/.
10Notably, MAIRA-2 use additional information at train and test time, including the radiologist-written prior report, when available.
Example 1
Reference: There are small bilateral pleural effusions with overlying atelectasis. No definite focal consolidation is seen. There is no pneumothorax. The aorta is calcified and tortuous. The cardiac silhouette is mildly enlarged.

Candidate report
CheXagent-2: There is a left basilar opacity silhouetting the hemidiaphragm. Elsewhere, the lungs are clear. The cardiome diastinal silhouette is stable. No acute osseous abnormality is identified.

CheXagent-2+DPO: The cardiac silhouette is enlarged but stable. There are small bilateral pleural effusions. There is a left basilar opacity which could be due to atelectasis although infection is not excluded. There is no pneumothorax. The mediastinal contours are stable.

Figure 4: Color-coded candidate reports. Green and red represent correct and incorrect. Orange highlights references to a prior imaging study.

the other two cases. In addition, GREEN score as Judge seems to be superior to BERTScore and we observe exact, and semantically equivalent, repetitions in some cases. Full results, along with an extensive discussion, are available in §C.4.

6 Conclusion

Our study highlights the significant potential of including preference fine-tuning in the post-training pipeline of medical VLMs. By developing an automated pipeline we effectively address the prohibitive cost of obtaining radiologist preferences at scale. Using our approach, we have shown that DAAs can substantially improve AI-generated radiology reports in clinically meaningful ways without additional radiologist feedback. Our approach achieves state-of-the-art performance on the MIMIC-CXR dataset while maintaining robust capabilities across diverse visual reasoning tasks. The surprising effectiveness of, inexpensive, general-domain NLG metrics for preference pair generation suggests a promising path forward for computationally efficient exploration of online alignment algorithms in this setting.

7 Limitations

Our work focuses on only two VLMs, CheXagent (Chen et al., 2024a) and CheXagent-2 (Chen et al., 2024b). While different in terms of underlying architecture, information used, and baseline performance, it would be of interest to additionally study VLMs from other families and sizes. In addition, our study of CheXagent-2 was more limited in nature and should be considered more preliminary.

Moreover, we treat clinically relevant metrics such as GREEN and CheXbert scores as the silver standard. While these metrics are highly relevant, a thorough evaluation involving clinical experts, radiologists, should be conducted for this study to be considered complete. Our qualitative analysis is a first step in this direction. However, conducting larger-scale reader studies remains an important direction for future work. We acknowledge potential biases beyond verbosity, such as societal biases related race, sex, or other demographic factors that may be embedded either in the underlying data or in the Judge. These potential biases warrant thorough investigation and mitigation in future work.

In addition, our hyperparameter search is non-exhaustive and it is possible that the relative ranking of the methods considered would change with a more extensive search.

Finally, we restrict ourselves to only offline DAAs. This leaves out a range of very competitive alignment algorithms, including on-policy RL algorithms, as well as the online, or iterative, counterparts to the DAAs considered. The recent success of DeepSeek-R1 (DeepSeek-AI et al., 2025) has led to a resurgence of interest in on-policy RL alignment algorithms. In particular, developing “verification functions” for the RRG task and employing reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2025) is an very interesting avenue for future work.

8 Acknowledgements

We would like to acknowledge compute support from Microsoft, Google, and Stanford Marlowe (Kapfer et al., 2025). D.H. is supported by MedTechLabs, the Göran Gustafsson foundation, and the foundation for Technical Scientific Research and Education. J.X. acknowledges support from the Canadian Institutes of Health Research (CIHR) and Oxford University Press. S.O. acknowledges funding from the DFG (German Research Foundation, ID: 517316550). M.V. is supported by graduate fellowship awards from the Department of Defense (NDSEG), the Knight-Hennessy Scholars program at Stanford University, and the Quad program. C.B. receives research support from the Promedica Foundation, Chur, CH. A.C. receives research support from NIH grants R01 HL167974, R01HL169345, R01 AR077604, R01 EB002524, R01 AR079431, P41 EB027060; ARPA-H grants AY2AX000045 and 1AYSAX0000024-01; and NIH contracts 75N92020C00008 and 75N92020C00021.
References

American College of Radiology. 2019. Siim-acr pneumothorax segmentation 2019.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A general language assistant as a laboratory for alignment. Preprint, arXiv:2112.00861.

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. Preprint, arXiv:2310.12036.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. Preprint, arXiv:2204.05862.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional ai: Harmlessness from ai feedback. Preprint, arXiv:2212.08073.

Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Anton Schwaighofer, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, Felix Meissen, Mercy Ranjit, Shaurya Srivastav, Julia Gong, Fabian Falck, Ozan Oktay, Anja Thiem, Matthew P. Lungren, Maria Teodora Wetscherek, Javier Alvarez-Valle, and Stephanie L. Hyland. 2024. Maira-2: Grounded radiology report generation. Preprint, arXiv:2406.04449.

Mythreyi Bhargavan, Jonathan H Sunshine, and Barbara Schepps. 2002. Too few radiologists? American Journal of Roentgenology, 178(5):1075–1082.

Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.

RJM Bruls and RM Kwee. 2020. Workload for radiologists during on-call hours: dramatic increase in the past 15 years. Insights into imaging, 11:1–7.

Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P Langlotz. 2024. CheXpert plus: Hundreds of thousands of aligned radiology texts, images and patients. arXiv preprint arXiv:2405.19538.

Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademli, Ziyi Yang, et al. 2024. Training small multimodal models to bridge biomedical competency gap: A case study in radiology imaging. arXiv preprint arXiv:2403.08002.

Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. 2024a. CheXagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208v1.

Zhihong Chen, Maya Varma, Justin Xu, Magdalini Paschali, Dave Van Veen, Andrew Johnston, Alaa Youssef, Louis Blankemeier, Christian Bluethgen, Stephan Altmaier, Jeya Maria Jose Valanarasu, Mohamed Siddig Eltayeb Munee, Eduardo Pontes Reis, Joseph Paul Cohen, Cameron Olsen, Tanishq Mathew Abraham, Emily B. Tsai, Christopher F. Beaulieu, Jenia Jitsev, Sergios Gaitidis, Jean-Benoit Delbrouck, Akshay S. Chaudhari, and Curtis P. Langlotz. 2024b. A vision-language foundation model to enhance efficiency of chest x-ray interpretation. Preprint, arXiv:2401.12208.

Yashin Dicente Cid, Matthew Macpherson, Louise Gervais-Andre, Yuanyi Zhu, Giuseppe Franco, Ruggero Santeramo, Chee Lim, Ian Selby, Keerthini Muthuswamy, Ashik Amlani, et al. 2024. Development and validation of open-source deep neural networks for comprehensive chest x-ray reading: a retrospective, multicentre study. The Lancet Digital Health, 6(1):e44–e57.

DeepSeek-AI. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang,
Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucang Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaqio Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangye Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxiang Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incarnitizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310.

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In Advances in Neural Information Processing Systems, volume 36, pages 30039–30069. Curran Associates, Inc.

Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple baseline of automatic evaluators. In First Conference on Language Modeling.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. Preprint, arXiv:2402.01306.

Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835–10866. PMLR.

Tarek N Hanna, Haris Shekhani, Christine Lamoureux, Hanna Mar, Refky Nicola, Clint Sliker, and Jamlik-Omari Johnson. 2017. Emergency radiology practice patterns: shifts, schedules, and job satisfaction. Journal of the American College of Radiology, 14(3):345–352.

Jiwoon Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. Preprint, arXiv:2403.07691.

Stephanie L Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Mercy Ranjit, Anton Schwaighofer, Fernando Pérez-García, Valentina Salvatelli, Shaury Srivastav, Anja Thieme, et al. 2023. Maira-1: A specialised large multimodal model for radiology report generation. arXiv preprint arXiv:2311.13668.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scão, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. Preprint, arXiv:2401.04088.

Haibo Jin, Haoxuan Che, Yi Lin, and Hao Chen. 2024. Promptmrg: Diagnosis-driven prompts for medical report generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(3):2607–2615.

Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-yung Deng, Roger G Mark, and Steven Horng. 2019. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317.

Craig Kapfer, Kurt Stine, Balasubramanian Narasimhan, Christopher Mentzel, and Emmanuel Candes. 2025. Marlowe: Stanford’s gpu-based computational instrument.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. 2025. Tulu 3: Pushing frontiers in open language model post-training. Preprint, arXiv:2411.15124.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024a. RLAI F vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 26874–26901. PMLR.

Suhyeon Lee, Won Jun Kim, Jinho Chang, and Jong Chul Ye. 2024b. LLM-CXR: Instruction-finetuned LLM for CXR image understanding and generation. In The Twelfth International Conference on Learning Representations.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024a. Visual instruction tuning. Advances in neural information processing systems, 36.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. Advances in neural information processing systems, 36.

Matthew Lyon, LaShon Sturgis, Darren Lendermon, Ann Marie Kuchinski, Taylor Mueller, Patrick Loeffler, Hongyan Xu, and Robert Gibson. 2015. Rural ed transfers due to lack of radiology services. The American journal of emergency medicine, 33(11):1630–1634.

World Health Organization et al. 2016. Communicating radiation risks in paediatric imaging: information to support health care discussions about benefit and risk.

Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluthgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. 2024. Green: Generative radiology report evaluation and error notation. In Findings of the Association for Computational Linguistics: EMNLP.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

WHO PAHO. 2012. World radiography day: Two-thirds of the world’s population has no access to diagnostic imaging. Pan American Health Organization.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pages 4998–5017, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Chantal Pellegrini, Matthias Keicher, Ege Özsoy, and Nassir Navab. 2023. Rad-restrict: A novel vqa benchmark and method for structured radiology reporting. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 409–419. Springer.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. 2024. Scaling laws for reward
model overoptimization in direct alignment algorithms. Preprint, arXiv:2406.02900.

Rafael Raffailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.

Abi Rimmer. 2017. Radiologist shortage leaves patient care at risk, warns royal college. BMJ: British Medical Journal (Online), 359.

Alexander T Ruutiainen, Daniel J Durand, Mary H Scanlon, and Jason N Itri. 2013. Increased error rates in preliminary reports issued by radiology residents working more than 10 consecutive hours overnight. Academic radiology, 20(3):305–311.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, et al. 2019. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1):e180041.

Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren. 2020. Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1500–1519.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. Preprint, arXiv:2303.15389.

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2024. Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110. Bangkok, Thailand. Association for Computational Linguistics.

Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Raffailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. 2024. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Forty-first International Conference on Machine Learning.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, et al. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.

Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. 2023. Xraypt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaerermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryuutar Tanno, Ira Ktena, Anil Palepu, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S. Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barrall, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, and Vivek Natarajan. 2024. Towards generalist biomedical ai. NEJM AI, 1(3):Aloa2300138.

R. J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256.

Ting Xiao, Lei Shi, Peng Liu, Zhe Wang, and Chen-jia Bai. 2024. Radiology report generation via multi-objective preference optimization. Preprint, arXiv:2412.08901.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc.

Yiyang Zhou, Chenhang Cui, Rafael Raffailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411.
Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, and Huaxiu Yao. 2025. Mmedpo: Aligning medical vision-language models with clinical-aware multimodal preference optimization. Preprint, arXiv:2412.06141.

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593.
A  RLHF and DAAs

RLHF (Ziegler et al., 2020; Stiennon et al., 2020; Ouyang et al., 2022) is based on the constrained reward maximization objective

\[
\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(y|x)} [R_\psi(x, y)] - \beta \mathbb{D}_{\mathrm{KL}} [\pi_\theta(y|x) || \pi_{\mathrm{ref}}(y|x)] ,
\]

where \( \mathbb{D}_{\mathrm{KL}} \) is the Kullback-Leibler (KL) divergence and \( \pi_{\mathrm{ref}} \) is the reference policy. \( R_\psi \) is the proxy reward model learned on a dataset of human preferences \( \mathcal{D} = \{ x^{(n)}, y_c^{(n)}, y_r^{(n)} \}_{n=1}^N \), where \( y_c \) and \( y_r \) denote the chosen and rejected completions for the prompt \( x \), such that \( y_c \succ y_r | x \).

Whilst extremely powerful, RLHF is computationally heavy, involves several steps, and can be tricky to implement in practice. Relatively recently, a new class of algorithms called DAAs (Rafailov et al., 2024) have become increasingly popular.\footnote{In this paper, we use this terminology more loosely than in Rafailov et al. (2024).} This class of algorithms re-parameterize the reward model via a change-of-variables using the closed-formed solution to the objective in (1), effectively bypassing both the reward modeling and reinforcement learning (RL) stages. Resulting in algorithms that remain performant yet computationally more light weight and easier to implement. DPO (Rafailov et al., 2023) was the first in this category and remains one of the most popular versions.

DPO exploits the closed-formed solution to (1), \( \pi(y|x) \propto \pi_{\mathrm{ref}}(y|x) \exp(R(x, y)/\beta) \) and the Bradley-Terry (BT) model (Bradley and Terry, 1952) of human preferences \( p^*(y_1 \succ y_2|x) = \sigma(\exp(R^*(x, y_1)) - \exp(R^*(x, y_2))) \), where \( R^* \) is the latent reward model, \( \exp \) is the exponential function, and \( \sigma \) is the logistic function. The reward can be isolated and written as a function of the policy \( R(x, y) = \beta \log \frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)} \). This re-parametrization can be applied to the latent reward \( R^* \) and substituted into the BT model, \( p^*(y_1 \succ y_1|x) = \sigma \left( \beta \log \frac{\pi^*(y_1|x)}{\pi_{\mathrm{ref}}(y_1|x)} - \beta \log \frac{\pi^*(y_2|x)}{\pi_{\mathrm{ref}}(y_2|x)} \right) \), where \( \pi^* \) is the optimal policy corresponding to the latent reward. Crucially, the probability of human preferences is now in terms of the policy instead of the reward model. A parameterized policy \( \pi_\theta \) can then be learned via a simple classification loss over the preference data

\[
\mathcal{L}_{\mathrm{DPO}}(\theta) = - \log \sigma \left( \beta \log \frac{\pi_\theta(y_c|x)}{\pi_{\mathrm{ref}}(y_c|x)} - \beta \log \frac{\pi_\theta(y_r|x)}{\pi_{\mathrm{ref}}(y_r|x)} \right).
\]

Hence, this change-of-variables has transformed a loss over rewards into a loss over policies.

B  Implementation Details

B.1  Data Details

We use the official train, validation, and test splits for the MIMIC-CXR (Johnson et al., 2019) and CheXpert Plus (Chambon et al., 2024) datasets. A key difference between CheXagent and CheXagent-2 is that CheXagent-2 additionally use indications, which provide clinical context, to aid in the RRG task. This means that CheXagent-2 imposes a stricter data requirement (i.e. each image-report pair must have a corresponding indication). Due to this, CheXagent-2 has slightly fewer examples than CheXagent for the RRG task. The number of examples in each split is available in Table 6. For MIMIC-CXR, the difference in test and validation is so small that it is negligible for the RRG task. On the other hand, for CheXpert Plus the difference is quite substantial. However, this is immaterial for any conclusions in this paper as we do not directly compare the models with each other on the CheXpert Plus data.

The view classification, binary image classification, single disease identification, multi disease identification, VQA, and image-text reasoning test data from the RSNA (Shih et al., 2019), SIIM (American College of Radiology, 2019), OpenI (Demner-Fushman et al., 2016), SLAKE (Liu et al., 2021), and Rad-Restruct (Pellegrini et al., 2023) datasets are processed as in Chen et al. (2024b).

B.2  Training Details

All models are trained using a machine with 4xA100, 4xA6000 or 8xH100 GPUs using learning rate learning rate \( 10^{-6} \). We set global batch size to 32 for CheXagent (Chen et al., 2024a) and 64 for
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="3">CheXagent</th>
    <th colspan="3">CheXagent-2</th>
  </tr>
  <tr>
    <th>Train</th>
    <th>Validation</th>
    <th>Test</th>
    <th>Train</th>
    <th>Validation</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>MIMIC-CXR</td>
    <td>148463</td>
    <td>1164</td>
    <td>2309</td>
    <td>148090</td>
    <td>1162</td>
    <td>2300</td>
  </tr>
  <tr>
    <td>CheXpert Plus</td>
    <td></td>
    <td></td>
    <td>208</td>
    <td></td>
    <td></td>
    <td>151</td>
  </tr>
</table>

Table 6: Number of examples in each split of MIMIC-CXR and CheXpert plus for CheXagent and CheXagent-2.

CheXagent-2 (Chen et al., 2024b). Each model is trained for one epoch. The image encoder is frozen while we train the LLM. Due to compute constraints, we only tune hyperparameters that are specific of the DAAs considered while keeping everything else fixed. An overview is given in Table 7, including optimal configurations. This non-exhaustive search was based on previous work and our initial experiments. For CheXagent, each \( \lambda \in [0.5, 1.0, 4.0, 5.0] \), ORPO resulted in a model which produced a special token at odd places, leading to a crash of our evaluation pipeline. We address this by catching the error and set the special token to the padding token. The same issue emerged for some \( \lambda \) for CheXagent-2.

<table>
  <tr>
    <th>Algorithm</th>
    <th>Objective</th>
    <th>Hyperparameters</th>
  </tr>
  <tr>
    <td>DPO (Rafailov et al., 2023)</td>
    <td>\( -\log \sigma \left( \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} - \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} \right) \)</td>
    <td>\( \beta \in [0.01, 0.05^*, 0.1] \)</td>
  </tr>
  <tr>
    <td>LC-DPO (Park et al., 2024)</td>
    <td>\( -\log \sigma \left( \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} - \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} + \alpha(|y_c| - |y_r|) \right) \)</td>
    <td>\( \beta \in [0.01, 0.05^*, 0.1], \alpha \in [0.001, 0.01^*] \)</td>
  </tr>
  <tr>
    <td>IPO (Azar et al., 2023)</td>
    <td>\( (\log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} - \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} - \frac{1}{2\tau})^2 \)</td>
    <td>\( \tau \in [0.1, 0.5, 1.0^*] \)</td>
  </tr>
  <tr>
    <td>KTO (Ethayarajh et al., 2024)</td>
    <td>\( -\lambda_c \sigma \left( \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} - z_{ref} \right) + \lambda_r \sigma \left( z_{ref} - \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)} \right) \)<br>where \( z_{ref} = \mathbb{E}_{(x,y)\sim D}[\beta \mathrm{KL}(\pi_\theta(y|x)||\pi_\theta(y|x))] \)</td>
    <td>\( \beta \in [0.01, 0.05^*, 0.1^*], \lambda_c = \lambda_r \)</td>
  </tr>
  <tr>
    <td>ORPO (Hong et al., 2024)</td>
    <td>\( -\log p_\theta(y|x) - \lambda \log \sigma \left[ \log \frac{\pi_{\theta}(y|x)}{1-p_{\theta}(y|x)} - \log \frac{\pi_{\theta}(y|x)}{1-p_{\theta}(y|x)} \right] \)<br>where \( p_\theta(y|x) = \exp \left( \frac{1}{|\eta|} \log \pi_{\theta}(y|x) \right) \)</td>
    <td>\( \lambda \in [0.5, 1.0, 4.0^*, 5.0^*] \)</td>
  </tr>
</table>

Table 7: Hyperparameter search for all direct alignment algorithms (DAAs) considered in this paper. For CheXagent, * and † denotes best for GREEN and the BERTScore as Judge, respectively. For CheXagent-2, \( \beta = 0.1 \) was found optimal for both GREEN and the BERTScore.

B.3 Sampling Details

In this paper, we set up CheXagent to treat the cases where the two CXRs are a frontal and a lateral image (i.e. from the same point in time) and a frontal and a prior frontal image (i.e. from two points in time) as separate cases with a separate prompt. This is in contrast to CheXagent-2, which employs the same prompt for both cases. Moreover, we employed stochastic sampling with temperature 1.0 for CheXagent at test time, while CheXagent-2 is based on greedy sampling, both with beam size set to 1. For preference pairs generation, both employ stochastic sampling with temperature 1.0 and beam size set to 1. These differences will explain parts of the performance differences between CheXagent and CheXagent-2. However, this is of minor concern to this study as we are mainly interested in the effect of preference fine-tuning, comparing the same setup prior and post alignment.

B.4 Evaluation Details

GREEN (Ostmeier et al., 2024) and GREEN-LC are based on the official implementation12 using StanfordAIMI/GREEN-radillama2-7b. The BERTScore (Zhang et al., 2019) used is from evaluate (v0.4.0) with distilbert-base-uncased as BERT model. The F1 CheXbert (Smit et al., 2020) scores use f1chexbert (v0.0.2) with default configurations. ROUGE (Lin, 2004) is from rouge-score (v0.1.2) using rougeL. Finally, the BLEU (Papineni et al., 2002) is BLEU-4, based on a custom code, available in the code base corresponding to this project.

CheXbert scores for GPT-4V13, MAIRA-1 (Hyland et al., 2023), Med-PaLM (Tu et al., 2024), and LLaVA-Rad (Chaves et al., 2024) are borrowed directly from Supplementary Table 1 in Chaves et al.

12https://github.com/Stanford-AIMI/GREEN.
13https://openai.com/index/gpt-4v-system-card/.
(2024). The CheXbert scores for MAIRA-2 (Bannur et al., 2024) are from Table D.1 in Bannur et al. (2024).

C Additional results

C.1 Preference data

Summary statistics for the chosen and rejected subsets are available in Table 8. We also report summary statistics of the length (in words) of the generated reports. Notably, the difference, or spread, in average length of the chosen and rejected subsets is slightly more pronounced for GREEN than for the BERTScore, 6.9% compared to 5.8% and much more significant overall for CheXagent-2 with a difference of 19.3% and 17.1%, respectively. To build further build intuition on the resulting chosen and rejected subsets we report CheXbert scores in Table 9. Scores for CheXagent and CheXagent-2 are not directly comparable since the training data is different. As expected, these scores indicate that the chosen subset is considerably better than the rejected. The average CheXbert scores for the chosen subset are more or less the same for GREEN and BERTScore as Judge. Similarly is true for the rejected subset, though now BERTScore as Judge results in slightly lower scores–meaning that the spread is marginally larger. Finally, as a qualitative study, we include a couple of examples of rejected and chosen candidates for CheXagent and CheXagent-2 is Fig. 5 and Fig. 6, respectively. We include the score assigned by the respective Judge in brackets.

<table>
  <tr>
    <th rowspan="2">CheXagent</th>
    <th colspan="6">GREEN</th>
    <th colspan="6">BERTScore</th>
  </tr>
  <tr>
    <th colspan="3">Metric</th>
    <th colspan="3">Report Length</th>
    <th colspan="3">Metric</th>
    <th colspan="3">Report Length</th>
  </tr>
  <tr>
    <th></th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
  </tr>
  <tr>
    <td>Chosen</td>
    <td>0.63</td>
    <td>0.60</td>
    <td>0.24</td>
    <td>56.3</td>
    <td>54.0</td>
    <td>20.2</td>
    <td>0.90</td>
    <td>0.90</td>
    <td>0.03</td>
    <td>55.1</td>
    <td>53.0</td>
    <td>20.2</td>
  </tr>
  <tr>
    <td>Rejected</td>
    <td>0.26</td>
    <td>0.22</td>
    <td>0.19</td>
    <td>52.7</td>
    <td>51.0</td>
    <td>24.9</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.04</td>
    <td>52.1</td>
    <td>50.0</td>
    <td>25.3</td>
  </tr>
  <tr>
    <th rowspan="2">CheXagent-2</th>
    <th colspan="6">Metric</th>
    <th colspan="6">Report Length</th>
  </tr>
  <tr>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
    <th>Mean</th>
    <th>Median</th>
    <th>Std.</th>
  </tr>
  <tr>
    <td>Chosen</td>
    <td>0.57</td>
    <td>0.50</td>
    <td>0.25</td>
    <td>54.4</td>
    <td>52.0</td>
    <td>19.3</td>
    <td>0.91</td>
    <td>0.90</td>
    <td>0.03</td>
    <td>52.7</td>
    <td>51.0</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>Rejected</td>
    <td>0.24</td>
    <td>0.20</td>
    <td>0.20</td>
    <td>45.6</td>
    <td>44.0</td>
    <td>22.8</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.05</td>
    <td>45.0</td>
    <td>43.0</td>
    <td>23.8</td>
  </tr>
</table>

Table 8: Summary statistics of reference-based metric and report length in the chosen and rejected subsets using GREEN and the BERTScore as Judges.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="5">Chosen</th>
    <th colspan="5">Rejected</th>
  </tr>
  <tr>
    <th colspan="2">F1-14</th>
    <th colspan="2">F1-5</th>
    <th colspan="2">F1-14</th>
    <th colspan="2">F1-5</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
    <th>Avg. (↑)</th>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
    <th>Avg. (↑)</th>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td>GREEN</td>
    <td>0.574</td>
    <td>0.668</td>
    <td>0.649</td>
    <td>0.700</td>
    <td>0.648</td>
    <td>0.411</td>
    <td>0.506</td>
    <td>0.455</td>
    <td>0.507</td>
    <td>0.463</td>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td>BERTScore</td>
    <td>0.582</td>
    <td>0.680</td>
    <td>0.647</td>
    <td>0.700</td>
    <td>0.652</td>
    <td>0.395</td>
    <td>0.488</td>
    <td>0.444</td>
    <td>0.493</td>
    <td>0.419</td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td>GREEN</td>
    <td>0.488</td>
    <td>0.587</td>
    <td>0.550</td>
    <td>0.621</td>
    <td>0.561</td>
    <td>0.336</td>
    <td>0.416</td>
    <td>0.365</td>
    <td>0.415</td>
    <td>0.373</td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td>BERTScore</td>
    <td>0.488</td>
    <td>0.594</td>
    <td>0.541</td>
    <td>0.615</td>
    <td>0.560</td>
    <td>0.316</td>
    <td>0.394</td>
    <td>0.346</td>
    <td>0.393</td>
    <td>0.362</td>
  </tr>
</table>

Table 9: CheXbert scores on the MIMIC-CXR on the chosen and rejected subsets.
Reference:
The heart size is normal. The mediastinal contours are normal. The hilar contours are normal. Bilateral atelectasis is present. There is no focal consolidation. There is no pleural effusion or pneumothorax. There is no free intraperitoneal air.

Chosen:
The lungs are well expanded, with no evidence of pleural effusion, pulmonary edema, pneumothorax or focal consolidation concerning for pneumonia. There is mild blunting of the costophrenic angle which may be due to atelectasis but a trace pleural effusion is not excluded. The cardiome diastal silhouette is stable. The partially imaged upper abdomen is unremarkable. (0.67)

Rejected:
The heart size is normal. Calcifications are noted at the aortic arch. The lungs are clear without focal consolidation. There is no pleural effusion or pneumothorax. There is no evidence of pulmonary edema. The air fluid levels seen on the most recent ct represent colonic interposition. (0.5)

Chosen:
Slightly forward patient rotation limits assessment. The heart size is normal. The mediastinal contours are normal. The hilar contours are normal. There is no focal consolidation, pleural effusion or pneumothorax. Bilateral opacities are most consistent with atelectasis. No displaced rib fracture is identified. (0.92)

Rejected:
The lungs are well expanded, with no evidence of pleural effusion, pulmonary edema, pneumothorax or focal consolidation concerning for pneumonia. There is mild blunting of the costophrenic angle which may be due to atelectasis but a trace pleural effusion is not excluded. The cardiome diastal silhouette is stable. The partially imaged upper abdomen is unremarkable. (0.89)

An endotracheal tube is in place, ending about 4.0 cm above the carina. An enteric tube is seen below the diaphragm, with its tip at the port at the jejunum and the distal tip in the proximal stomach. Recommend advancement of the enteric tube to better placement within the stomach. A right internal jugular central venous catheter is present, ending in the mid-to-lower SVC without pneumothorax. There are extensive left greater than right bilateral alveolar opacities which continue to increase. Differential diagnosis: pulmonary hemorrhage with quickly moving infection, pulmonary hemorrhage, ards, component of underlying esophageal is not excluded. Possible left pleural effusion is seen.

An endotracheal tube is in place, ending about 4.9 cm above the carina. An enteric tube is seen, ending in the distal esophagus. A right internal jugular central venous catheter is present, ending in the mid svc. There are extensive left greater than right bilateral airspace opacities. A small left pleural effusion is seen. No pneumothorax is seen. The cardiac silhouette is stable. The mediastinal silhouette is stable. (0.38)

An endotracheal tube is in a stable position. An interval placement of a right iJ has been done and appears to be in the correct position. There are extensive left greater than right bilateral alveolar opacities which may be due to pulmonary hemorrhage contusion. Mildly displaced left sided rib fractures are seen. (0.17)

An endotracheal tube is in place, ending about 4.9 cm above the carina. An enteric tube is seen, ending in the distal esophagus. A right internal jugular central venous catheter is present, ending in the mid svc. There are extensive left greater than right bilateral airspace opacities. A small left pleural effusion is seen. No pneumothorax is seen. The cardiac silhouette is stable. The mediastinal silhouette is stable. (0.90)

An endotracheal tube is in a stable position. An interval placement of a right iJ has been done and appears to be in the correct position. There are extensive left greater than right bilateral alveolar opacities which may be due to pulmonary hemorrhage contusion. Mildly displaced left sided rib fractures are seen. (0.85)

Figure 5: First two examples of chosen and rejected candidates for CheXagent in the MIMIC-CXR train set. Number in brackets is assigned score (GREEN or BERTScore).
Reference:
There is no focal consolidation. There is no pleural effusion or pneumothorax. Bilateral nodular opacities are most likely represent nipple shadows. The cardiome diastal silhouette is normal. Clips project over the left lung, potentially within the chest wall. The imaged upper abdomen is unremarkable. Chronic deformity of the posterior left sixth and seventh ribs are noted.

Chosen:
The lungs are clear without focal consolidation, effusion or edema. The cardiome diastal silhouette is within normal limits. Chronic abnormalities of the left fifth rib are unchanged. Surgical clips are seen overlying the periphery of the left lung and anterior chest wall. No acute osseous abnormalities. (0.6)

Rejected:
Surgical clips are seen over the left breast. There is mild streaky density on the left, likely due to subsegmental atelectasis. Nipple shadows are visible bilaterally. The lungs appear clear without focal consolidation. The cardiome diastal silhouette is normal. (0.29)

Chosen:
There is no focal consolidation, effusion, or pneumothorax. The cardiome diastal silhouette is normal. The imaged osseous structures are intact. No free air is seen below the right hemidiaphragm and no surgical clips in the left chest wall (0.9)

Rejected:
The lungs are clear of consolidation, effusion, or vascular congestion. The cardiome diastal silhouette is within normal limits. Surgical clips project over the left chest wall and right atrial. (0.88)

The lung volumes are low. The cardiac contours appear unchanged. The mediastinal contours appear unchanged. The hilar contours appear unchanged. There are a number of round nodular densities projecting over each lung, ranging from numerous and discretely visualized in the left upper lobe. However, in addition, there is a more hazy widespread opacity projecting over the mid upper lung which could be compatible with a coinciding pneumonia. Pulmonary nodules in the left upper lobe are not completely characterized in this study. There is no pleural effusion or pneumothorax. Post-operative changes are similar along the right chest wall.

Lung volumes are low. The cardiac contours appear stable. The mediastinal contours appear stable. The hilar contours appear stable. There is no definite signs of effusion or pneumothorax. The lungs demonstrate perihilar ground-glass opacities with an upper lobe predominance, left greater than right. These suggest large airways inflammation or infection. There is a nodular focus projecting over the left upper lobe measuring up to 8mm, perhaps focus of acute infection, inflammatory or potentially a pulmonary nodule. Small nodular focus projecting over the right costophrenic angle suggests a stable nodule measuring 6 mm or less. Mild-to-moderate bilateral atelectasis appears unchanged on the left and improved on the right. (0.38)

There are multiple bilateral pulmonary nodules. There are no definite signs of pneumonia in the lower lungs. There is no pleural effusion or pneumothorax. The cardiome diastal silhouette is stable. The bony structures are intact. (0.18)

Lung volumes are low. The cardiac contours are within normal limits. The mediastinal contours are within normal limits. The hilar contours are within normal limits. There is crowding of the bronchovascular structures but no overt pulmonary edema is demonstrated. Multiple nodular opacities are present bilaterally, being more numerous on the left. The lung volumes are low. Patchy opacities are noted in both lung bases, more pronounced within the left lung base. Chronic fibrosis thickening within the right lung base is observed, with blunting of the lateral costophrenic angle compatible with chronic pleural thickening. No pneumothorax or large pleural effusion is demonstrated. There are no acute osseous abnormalities. (0.90)

Multiple bilateral lung nodules are present without obvious cavitation. There is increased bibasilar atelectasis. The cardiome diastal silhouette and hilar contours remain near normal allowing for low lung volumes. There is no large pleural effusion or pneumothorax. Chronic right rib deformities are unchanged. (0.58)

Figure 6: First two examples of chosen and rejected candidates for CheXagent-2 in the MIMIC-CXR train set. Number in brackets is assigned score (GREEN or BERTScore).
C.2 Judge Optimization Results

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="6">MIMIC-CXR</th>
  </tr>
  <tr>
    <th>GREEN (↑)</th>
    <th>GREEN-LC (↑)</th>
    <th>BERTScore(↑)</th>
    <th>BLEU-4 (↑)</th>
    <th>ROUGE-L (↑)</th>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td></td>
    <td>0.249</td>
    <td>0.218</td>
    <td>0.856</td>
    <td>0.041</td>
    <td>0.274</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.325 (30.6)</td>
    <td>0.263 (20.5)</td>
    <td>0.862 (0.65)</td>
    <td>0.057 (39.9)</td>
    <td>0.293 (6.94)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.320 (28.7)</td>
    <td>0.288 (32.0)</td>
    <td>0.864 (0.90)</td>
    <td>0.056 (38.4)</td>
    <td>0.302 (10.4)</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>GREEN</td>
    <td>0.326 (30.9)</td>
    <td>0.282 (29.3)</td>
    <td>0.863 (0.84)</td>
    <td>0.059 (45.0)</td>
    <td>0.297 (8.57)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td><b>0.328 (31.9)</b></td>
    <td><b>0.293 (34.1)</b></td>
    <td>0.867 (1.27)</td>
    <td>0.056 (38.8)</td>
    <td>0.305 (11.4)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.322 (29.4)</td>
    <td>0.275 (26.2)</td>
    <td>0.862 (0.69)</td>
    <td>0.053 (29.5)</td>
    <td>0.290 (6.08)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.285 (14.4)</td>
    <td>0.242 (11.0)</td>
    <td>0.869 (1.56)</td>
    <td>0.059 (44.2)</td>
    <td>0.309 (12.9)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.283 (13.5)</td>
    <td>0.259 (18.7)</td>
    <td>0.871 (1.75)</td>
    <td>0.057 (41.2)</td>
    <td><b>0.315 (15.1)</b></td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>BERTScore</td>
    <td>0.283 (13.6)</td>
    <td>0.247 (13.1)</td>
    <td>0.870 (1.62)</td>
    <td><b>0.060 (47.2)</b></td>
    <td>0.309 (12.9)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>0.304 (21.9)</td>
    <td>0.279 (28.2)</td>
    <td><b>0.872 (1.88)</b></td>
    <td>0.056 (37.7)</td>
    <td>0.310 (13.4)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.291 (16.9)</td>
    <td>0.265 (21.4)</td>
    <td>0.869 (1.57)</td>
    <td>0.054 (33.8)</td>
    <td>0.307 (12.0)</td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td></td>
    <td>0.326</td>
    <td>0.297</td>
    <td>0.888</td>
    <td>0.136</td>
    <td>0.350</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>GREEN</td>
    <td><b>0.387 (18.9)</b></td>
    <td><b>0.339 (14.1)</b></td>
    <td>0.891 (0.301)</td>
    <td>0.150 (10.3)</td>
    <td>0.357 (1.93)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>BERTScore</td>
    <td>0.352 (8.11)</td>
    <td>0.326 (9.58)</td>
    <td><b>0.896 (0.949)</b></td>
    <td><b>0.153 (12.5)</b></td>
    <td><b>0.372 (6.37)</b></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="6">CheXpert Plus</th>
  </tr>
  <tr>
    <th>GREEN (↑)</th>
    <th>GREEN-LC (↑)</th>
    <th>BERTScore(↑)</th>
    <th>BLEU-4 (↑)</th>
    <th>ROUGE-L (↑)</th>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td></td>
    <td>0.248</td>
    <td>0.202</td>
    <td>0.851</td>
    <td>0.038</td>
    <td>0.274</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.322 (29.9)</td>
    <td>0.222 (9.69)</td>
    <td>0.859 (0.91)</td>
    <td>0.053 (40.2)</td>
    <td>0.289 (5.25)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.330 (32.8)</td>
    <td><b>0.268 (32.3)</b></td>
    <td>0.861 (1.26)</td>
    <td>0.049 (30.2)</td>
    <td>0.305 (10.6)</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>GREEN</td>
    <td>0.334 (34.6)</td>
    <td>0.267 (31.9)</td>
    <td>0.861 (1.15)</td>
    <td>0.052 (39.1)</td>
    <td>0.295 (7.22)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td><b>0.341 (37.2)</b></td>
    <td>0.266 (31.4)</td>
    <td>0.863 (1.42)</td>
    <td>0.052 (36.9)</td>
    <td>0.305 (10.9)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.326 (31.4)</td>
    <td>0.232 (14.6)</td>
    <td>0.856 (0.64)</td>
    <td>0.046 (21.9)</td>
    <td>0.287 (4.63)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.278 (12.0)</td>
    <td>0.208 (2.84)</td>
    <td>0.867 (1.89)</td>
    <td>0.054 (43.6)</td>
    <td>0.307 (11.3)</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.299 (20.3)</td>
    <td>0.248 (22.5)</td>
    <td><b>0.871 (2.34)</b></td>
    <td><b>0.056 (47.6)</b></td>
    <td><b>0.318 (15.1)</b></td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>BERTScore</td>
    <td>0.282 (13.6)</td>
    <td>0.218 (7.88)</td>
    <td>0.866 (1.75)</td>
    <td>0.052 (37.6)</td>
    <td>0.309 (12.0)</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>0.308 (24.0)</td>
    <td>0.249 (23.1)</td>
    <td>0.869 (2.11)</td>
    <td>0.054 (43.8)</td>
    <td>0.316 (14.6)</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.298 (19.9)</td>
    <td>0.240 (18.4)</td>
    <td>0.870 (2.21)</td>
    <td>0.049 (30.8)</td>
    <td>0.314 (14.0)</td>
  </tr>
  <tr>
    <td>CheXagent-2-3b</td>
    <td></td>
    <td>0.349</td>
    <td>0.304</td>
    <td>0.892</td>
    <td>0.123</td>
    <td>0.361</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>GREEN</td>
    <td><b>0.387 (10.89)</b></td>
    <td><b>0.320 (5.34)</b></td>
    <td>0.888 (-0.37)</td>
    <td>0.126 (2.74)</td>
    <td>0.356 (-1.30)</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td>BERTScore</td>
    <td>0.359 (2.88)</td>
    <td>0.310 (1.78)</td>
    <td><b>0.893 (0.15)</b></td>
    <td><b>0.130 (2.86)</b></td>
    <td><b>0.357 (-0.91)</b></td>
  </tr>
</table>

Table 10: Results on the MIMIC-CXR and CheXpert Plus test sets (percentage change compared to SFT baseline in brackets). Best results in bold, separate for CheXagent and CheXagent-2.
C.3  Generalization to CheXbert Scores

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="2">F1-14</th>
    <th colspan="2">F1-5</th>
    <th rowspan="2">Avg. (↑)</th>
  </tr>
  <tr>
    <th>Macro (↑)</th>
    <th>Micro (↑)</th>
    <th>Macro(↑)</th>
    <th>Micro (↑)</th>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td></td>
    <td>20.4</td>
    <td>35.5</td>
    <td>19.6</td>
    <td>25.8</td>
    <td>25.3</td>
  </tr>
  <tr>
    <td>MAIRA-1</td>
    <td></td>
    <td>38.6</td>
    <td>55.7</td>
    <td>47.7</td>
    <td>56.0</td>
    <td>49.5</td>
  </tr>
  <tr>
    <td>MAIRA-2</td>
    <td></td>
    <td>41.6</td>
    <td>58.1</td>
    <td>50.4</td>
    <td>59.1</td>
    <td>52.3</td>
  </tr>
  <tr>
    <td>Med-PaLM M (12B)</td>
    <td></td>
    <td>37.3</td>
    <td>51.4</td>
    <td>50.6</td>
    <td>56.5</td>
    <td>49.0</td>
  </tr>
  <tr>
    <td>Med-PaLM M (84B)</td>
    <td></td>
    <td>39.8</td>
    <td>53.6</td>
    <td>51.6</td>
    <td>57.9</td>
    <td>50.7</td>
  </tr>
  <tr>
    <td>Med-PaLM M (562B)</td>
    <td></td>
    <td>37.8</td>
    <td>51.6</td>
    <td>49.9</td>
    <td>56.3</td>
    <td>48.9</td>
  </tr>
  <tr>
    <td>LLaVA-Rad</td>
    <td></td>
    <td>39.5</td>
    <td>57.3</td>
    <td>47.7</td>
    <td>57.4</td>
    <td>50.5</td>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td></td>
    <td>38.9<sub>[37.8, 40.0]</sub></td>
    <td>50.9<sub>[50.0, 51.8]</sub></td>
    <td>47.6<sub>[46.3, 48.8]</sub></td>
    <td>54.1<sub>[52.9, 55.3]</sub></td>
    <td>47.9<sub>[46.9, 48.8]</sub></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>41.5<sub>[40.2, 42.9]</sub></td>
    <td>54.1<sub>[53.2, 54.9]</sub></td>
    <td>51.8<sub>[50.4, 53.1]</sub></td>
    <td>58.3<sub>[57.1, 59.5]</sub></td>
    <td>51.4<sub>[50.4, 52.4]</sub></td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>37.6<sub>[36.3, 38.9]</sub></td>
    <td>52.1<sub>[51.1, 53.1]</sub></td>
    <td>47.5<sub>[46.1, 49.0]</sub></td>
    <td>55.2<sub>[53.9, 56.4]</sub></td>
    <td>48.1<sub>[47.1, 49.2]</sub></td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td></td>
    <td>39.0<sub>[37.7, 40.3]</sub></td>
    <td>52.9<sub>[52.0, 53.8]</sub></td>
    <td>48.9<sub>[47.5, 50.4]</sub></td>
    <td>56.5<sub>[55.3, 57.7]</sub></td>
    <td>49.3<sub>[48.3, 50.3]</sub></td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>40.7<sub>[39.4, 41.9]</sub></td>
    <td>55.0<sub>[54.1, 55.9]</sub></td>
    <td>51.6<sub>[50.3, 52.9]</sub></td>
    <td>59.3<sub>[58.1, 60.4]</sub></td>
    <td>51.7<sub>[50.7, 52.6]</sub></td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>41.4<sub>[40.1, 42.7]</sub></td>
    <td>55.0<sub>[54.2, 55.9]</sub></td>
    <td>51.5<sub>[50.1, 52.8]</sub></td>
    <td>58.3<sub>[57.1, 59.5]</sub></td>
    <td>51.5<sub>[50.8, 52.5]</sub></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>42.5<sub>[41.4, 43.7]</sub></td>
    <td>56.1<sub>[55.2, 56.9]</sub></td>
    <td>54.7<sub>[53.5, 55.9]</sub></td>
    <td>61.6<sub>[60.6, 62.6]</sub></td>
    <td>53.7<sub>[52.8, 54.6]</sub></td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>42.3<sub>[41.0, 43.4]</sub></td>
    <td>56.3<sub>[55.5, 57.2]</sub></td>
    <td>52.7<sub>[51.3, 54.0]</sub></td>
    <td>60.8<sub>[59.6, 61.9]</sub></td>
    <td>53.2<sub>[52.1, 54.0]</sub></td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td></td>
    <td>43.1<sub>[41.9, 44.2]</sub></td>
    <td>57.0<sub>[56.2, 57.8]</sub></td>
    <td>53.6<sub>[52.3, 54.8]</sub></td>
    <td>61.5<sub>[60.4, 62.5]</sub></td>
    <td>53.8<sub>[52.9, 54.6]</sub></td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>44.0<sub>[42.9, 45.3]</sub></td>
    <td>58.0<sub>[57.2, 58.9]</sub></td>
    <td>54.0<sub>[52.8, 55.2]</sub></td>
    <td>62.3<sub>[61.2, 63.4]</sub></td>
    <td>54.6<sub>[53.7, 55.5]</sub></td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>42.4<sub>[41.2, 43.7]</sub></td>
    <td>56.7<sub>[55.9, 57.6]</sub></td>
    <td>52.5<sub>[51.3, 53.6]</sub></td>
    <td>60.7<sub>[59.7, 61.7]</sub></td>
    <td>53.1<sub>[52.2, 53.9]</sub></td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td></td>
    <td>44.6<sub>[43.4, 45.9]</sub></td>
    <td>57.8<sub>[56.9, 58.7]</sub></td>
    <td>55.5<sub>[54.2, 57.1]</sub></td>
    <td>62.4<sub>[61.2, 63.6]</sub></td>
    <td>55.1<sub>[54.1, 56.1]</sub></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td><b>45.8</b><sub>[44.5, 47.1]</sub></td>
    <td><b>59.7</b><sub>[58.9, 60.6]</sub></td>
    <td><b>56.0</b><sub>[54.6, 57.5]</sub></td>
    <td><b>64.1</b><sub>[63.0, 65.3]</sub></td>
    <td><b>56.6</b><sub>[55.4, 57.4]</sub></td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>43.0<sub>[41.7, 44.4]</sub></td>
    <td>59.7<sub>[58.8, 60.6]</sub></td>
    <td>53.5<sub>[52.1, 54.9]</sub></td>
    <td>63.8<sub>[62.6, 65.0]</sub></td>
    <td>55.0<sub>[54.1, 56.0]</sub></td>
  </tr>
</table>

Table 11: CheXbert scores on the MIMIC-CXR test set. 90% confidence interval obtained by 1000 bootstrap samples in subscripts. Best results in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Judge</th>
    <th colspan="14">F1-scores (↑)</th>
  </tr>
  <tr>
    <th>ECm.</th>
    <th>Cmgl.</th>
    <th>LOpac.</th>
    <th>LLes.</th>
    <th>Edema</th>
    <th>Cnsl.</th>
    <th>Pna.</th>
    <th>Atel.</th>
    <th>PEff.</th>
    <th>Pmtx.</th>
    <th>Poth.</th>
    <th>Frac.</th>
    <th>SuDev.</th>
    <th>NoF.</th>
  </tr>
  <tr>
    <td>CheXagent</td>
    <td></td>
    <td>0.347</td>
    <td>0.620</td>
    <td>0.461</td>
    <td>0.171</td>
    <td>0.493</td>
    <td>0.158</td>
    <td>0.227</td>
    <td>0.453</td>
    <td>0.655</td>
    <td>0.444</td>
    <td><b>0.992</b></td>
    <td><b>0.240</b></td>
    <td>0.787</td>
    <td>0.304</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.372</td>
    <td>0.675</td>
    <td>0.444</td>
    <td>0.186</td>
    <td>0.525</td>
    <td>0.226</td>
    <td>0.231</td>
    <td>0.452</td>
    <td>0.710</td>
    <td>0.416</td>
    <td>0.178</td>
    <td>0.217</td>
    <td>0.822</td>
    <td>0.365</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.385</td>
    <td>0.666</td>
    <td>0.380</td>
    <td>0.178</td>
    <td>0.430</td>
    <td>0.211</td>
    <td>0.158</td>
    <td>0.404</td>
    <td>0.665</td>
    <td>0.407</td>
    <td>0.083</td>
    <td>0.129</td>
    <td>0.835</td>
    <td>0.339</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>GREEN</td>
    <td>0.402</td>
    <td>0.686</td>
    <td>0.395</td>
    <td>0.160</td>
    <td>0.472</td>
    <td>0.222</td>
    <td>0.206</td>
    <td>0.405</td>
    <td>0.661</td>
    <td>0.483</td>
    <td>0.090</td>
    <td>0.138</td>
    <td>0.822</td>
    <td>0.321</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td>0.388</td>
    <td>0.682</td>
    <td>0.422</td>
    <td>0.202</td>
    <td>0.542</td>
    <td>0.192</td>
    <td>0.180</td>
    <td>0.444</td>
    <td>0.721</td>
    <td>0.552</td>
    <td>0.074</td>
    <td>0.118</td>
    <td><b>0.841</b></td>
    <td>0.349</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.348</td>
    <td>0.684</td>
    <td>0.479</td>
    <td>0.201</td>
    <td>0.492</td>
    <td>0.224</td>
    <td><b>0.247</b></td>
    <td>0.475</td>
    <td>0.698</td>
    <td>0.511</td>
    <td>0.072</td>
    <td>0.177</td>
    <td>0.835</td>
    <td>0.365</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.373</td>
    <td>0.688</td>
    <td>0.467</td>
    <td>0.192</td>
    <td><b>0.580</b></td>
    <td><b>0.236</b></td>
    <td>0.141</td>
    <td>0.503</td>
    <td>0.728</td>
    <td>0.483</td>
    <td>0.157</td>
    <td>0.239</td>
    <td>0.831</td>
    <td>0.336</td>
  </tr>
  <tr>
    <td>+LC-DPO</td>
    <td></td>
    <td>0.301</td>
    <td>0.692</td>
    <td><b>0.506</b></td>
    <td>0.208</td>
    <td>0.567</td>
    <td>0.175</td>
    <td>0.107</td>
    <td>0.471</td>
    <td>0.731</td>
    <td>0.567</td>
    <td>0.167</td>
    <td>0.225</td>
    <td>0.832</td>
    <td>0.376</td>
  </tr>
  <tr>
    <td>+IPO</td>
    <td>BERTScore</td>
    <td>0.386</td>
    <td>0.690</td>
    <td>0.500</td>
    <td><b>0.218</b></td>
    <td>0.559</td>
    <td>0.190</td>
    <td>0.111</td>
    <td>0.500</td>
    <td><b>0.739</b></td>
    <td><b>0.530</b></td>
    <td><b>0.195</b></td>
    <td>0.220</td>
    <td>0.836</td>
    <td>0.342</td>
  </tr>
  <tr>
    <td>+KTO</td>
    <td></td>
    <td><b>0.430</b></td>
    <td><b>0.701</b></td>
    <td>0.487</td>
    <td>0.217</td>
    <td>0.577</td>
    <td>0.172</td>
    <td>0.201</td>
    <td><b>0.514</b></td>
    <td>0.735</td>
    <td><b>0.584</b></td>
    <td>0.166</td>
    <td>0.197</td>
    <td>0.830</td>
    <td>0.360</td>
  </tr>
  <tr>
    <td>+ORPO</td>
    <td></td>
    <td>0.377</td>
    <td>0.690</td>
    <td>0.491</td>
    <td>0.148</td>
    <td>0.562</td>
    <td>0.157</td>
    <td>0.231</td>
    <td>0.481</td>
    <td>0.736</td>
    <td>0.497</td>
    <td>0.116</td>
    <td>0.218</td>
    <td>0.826</td>
    <td><b>0.418</b></td>
  </tr>
  <tr>
    <td>CheXagent-2</td>
    <td></td>
    <td>0.324</td>
    <td>0.672</td>
    <td>0.539</td>
    <td><b>0.244</b></td>
    <td>0.606</td>
    <td><b>0.235</b></td>
    <td>0.284</td>
    <td>0.543</td>
    <td>0.720</td>
    <td><b>0.527</b></td>
    <td>0.085</td>
    <td><b>0.330</b></td>
    <td>0.815</td>
    <td>0.333</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td><b>0.450</b></td>
    <td>0.696</td>
    <td>0.493</td>
    <td>0.218</td>
    <td>0.606</td>
    <td>0.214</td>
    <td><b>0.325</b></td>
    <td><b>0.545</b></td>
    <td>0.738</td>
    <td>0.526</td>
    <td><b>0.157</b></td>
    <td><b>0.255</b></td>
    <td><b>0.856</b></td>
    <td>0.343</td>
  </tr>
  <tr>
    <td>+DPO</td>
    <td></td>
    <td>0.358</td>
    <td><b>0.703</b></td>
    <td><b>0.547</b></td>
    <td>0.166</td>
    <td><b>0.622</b></td>
    <td>0.127</td>
    <td>0.066</td>
    <td>0.478</td>
    <td><b>0.749</b></td>
    <td>0.550</td>
    <td>0.143</td>
    <td>0.271</td>
    <td>0.853</td>
    <td><b>0.396</b></td>
  </tr>
</table>

Table 12: F1 scores on the MIMIC-CXR test set using 14 categories from the CheXbert labeler (Smit et al., 2020): Enlarged Cardiomegastinum (ECm.), Cardiomegaly (Cmgl.), Lung Opacity (LOpac.), Lung Lesion (LLes.), Edema, Consolidation (Cnsl.), Pneumonia (Pna.), Atelectasis (Atel.), Pleural Effusion (PEff.), Pneumothorax (Pmtx.), Pleural Other (POth.), Fracture (Frac.), Support Devices (SuDev.), no Findings (NoF.). Best results in bold, separate for CheXagent and CheXagent-2.
C.4 Qualitative Analysis
To build some further intuition of behavior pre and post alignment, we conduct a qualitative study in which a board-certified radiologist was asked to color-code candidate reports from CheXagent, CheXagent-2, as well as their top performing aligned versions for GREEN and BERTScore as Judge, respectively, in terms of cheXbert scores in Table 4. We selected three interesting, and representative, examples from MIMIC-CXR test set, including examples with positive and negative findings, only negative findings, and with and without indications. These were subsequently given to a radiologist to color-code, using the reference reports as well as the CXRs in png format (i.e. exact measurements are not possible). Three cases with six reports each, resulted in a total of 18 reports to be processed. The reports are color-coded as green (correct), red (incorrect), and orange to indicate where an reference to prior report is made though none was presented at the time of inference. For instance, the statement that something is “stable” may or may not be true, depending on the prior imaging study.

Example 1 and 2 are available in Fig. 7. For example 1, CheXagent-2, despite being a very strong model, contains two errors. First, it incorrectly states that lungs are elsewhere clear—marked in red. Secondly, comparing with the reference report, the detection of small bilateral pleural effusions is missing. None of these errors are made by the aligned versions, using both GREEN and BERTScore as Judge. Moreover, for CheXagent with both GREEN and BERTScore as Judge, there are repetitions. For GREEN “no overt edema” is mentioned twice. For BERTScore as Judge, “no pneumothorax” is repeated. These types of exact repetitions may be due to verbosity bias.

Moving on to example 2, we can see that there are exact repetition for CheXagent-2 aligned with DPO. “There is no pneumothorax” and “There is no pleural effusion” are bot repeated as “There is no evidence of ...”. Interestingly, such repetition are not present for BERTScore as Judge, nor for the results from CheXagent. Notably, however, both for CheXagent-2 and CheXagent there are errors in the candidate report using BERTScore as Judge for this case. “Support devices have been removed” from CheXagent cannot cannot be supported nor refuted definitively.

Finally, consider example 3 in Fig. 8. This case includes a lot of details, with a reference report significantly longer than the previous two cases. One thing that is very interesting about this case is that the reference report includes a quantitative measurement: the distance between where the endotracheal tube terminates and the carina, expressed in centimeters. Quantitative results are given in some of the candidate reports. However, due to the set up of this reader study (i.e. that we used CXRs in png format) it is not feasible for the radiologist to make measurements and determine distance definitively. To capture this additional feature, we introduce the additional color blue. Takeaways are similar as to the previous two cases. Errors are made by CheXagent-2 and CheXagent, as well as the aligned versions using BERTScore as Judge. For GREEN as Judge, the performance remains strong, with no direct errors. Moreover, cheXagent makes a partial mistake not capture by our color coding scheme in this case as it states that there is “A small right pleural effusion,” where it is actually bilateral.
Example 1

Reference: There are small bilateral pleural effusions with overlying atelectasis. No definite focal consolidation is seen. There is no pneumothorax. The aorta is calcified and tortuous. The cardiac silhouette is mildly enlarged.

Candidate reports

CheXAgent-2: There is a left basilar opacity silhouetting the hemidiaphragm. Elsewhere, the lungs are clear. The cardiome diastinal silhouette is stable. No acute osseous abnormality is identified.

CheXAgent-2 + DPO (GREEN): The cardiac silhouette is enlarged but stable. There are small bilateral pleural effusions. There is a left basilar opacity which could be due to atelectasis although infection is not excluded. There is no pneumothorax. The mediastinal contours are stable.

CheXAgent-2 + DPO (BERTScore): There is left basilar opacity. Small bilateral pleural effusions are present. The lungs are otherwise clear. The cardiac silhouette is enlarged. The mediastinal contours are stable. There is no pneumothorax.

CheXAgent: The heart is at the upper limits of normal size. The mediastinal contour is prominent likely due to an unfolded thoracic aorta. The lungs are well expanded with left basilar linear opacity, likely representing atelectasis. There is no focal consolidation concerning for pneumonia. There may be tiny bilateral peripheral nodules. No pneumothorax is detected. There is mild rightward convex curvature centered along the mid thoracic spine with small to moderate osteophyte formation.

CheXAgent + KTO (GREEN): There are tiny, persistent bilateral pleural effusions. There is no overt edema or pneumothorax. The cardiome diastinal silhouette is stable. There is no mediastinal widening. There are no definite signs of pneumonia. There is no overt edema. The imaged osseous structures appear intact.

CheXAgent + KTO (BERTScore): There are small bilateral pleural effusions with overlying mild atelectasis. The cardiac silhouette is mildly enlarged. The cardiome diastinal silhouette is stable. There is no pneumothorax. Tortuosity of the descending thoracic aorta is noted. There is no pneumothorax. No acute osseous abnormalities are seen.

Example 2

Indication: ...-year-old male with esophageal stricture, now status post esophageal dilation and Kenalog injection, requiring assessment for pneumomediastinum and pneumothorax.

Reference: There is no pneumothorax or pneumomediastinum. There is no deep cervical air. The lungs are well expanded. There is no evidence of acute cardiac or pulmonary process. The cardiome diastinal silhouette is unremarkable.

Candidate reports

CheXAgent-2: The lungs are well expanded. There is a right lower lobe opacity. There is no pneumothorax or pleural effusion. The heart is top normal in size. The mediastinal contours are normal. The hilar contours are normal. There is a mild dextroscoliosis of the thoracic spine.

CheXAgent-2 + DPO (GREEN): The lungs are well expanded. There is no focal consolidation. There is no pneumothorax. There is no pleural effusion. The heart size is normal. The mediastinal silhouette is normal. The hilar contours are normal. There is no evidence of pneumomediastinum. There is no evidence of pneumothorax. There is no evidence of pleural effusion.

CheXAgent-2 + DPO (BERTScore): The lungs are well expanded. There is mild right basilar atelectasis. There is no pneumothorax. The cardiome diastinal silhouette is normal. The hilar contours are normal. There is no pleural effusion.

CheXAgent: The heart size is normal. The hilar and mediastinal contours are within normal limits. Support devices have been removed and lung volumes are improved. The opacity in the right lower lung has improved and may represent atelectasis. There is no opacity, pulmonary edema or pneumothorax. A small right pleural effusion is seen.

CheXAgent + KTO (GREEN): There is no pleural effusion or pneumothorax. There is no pulmonary edema. The heart size is at the upper limit of normal. There is no focal consolidation concerning for aspiration or pneumonia. The hilar and mediastinal silhouettes are normal.

CheXAgent + KTO (BERTScore): The heart size is normal. The cardiome diastinal silhouette and hilar contours are normal. There is minimal opacification at the right lung base. The lungs are otherwise clear. There is no pleural effusion or pneumothorax.

Figure 7: Color-coded candidate reports: Examples 1 and 2. Green and red represent correct and incorrect. Orange refers to prior imaging study.
Example 3

Reference: The endotracheal tube terminates 4.2 cm above the carina. The side port of a nasogastric tube is below the diaphragm. Pulmonary vasculature is ill-defined, compatible with severe pulmonary edema. Hazy opacity overlying both lungs and blunting of the costophrenic angles are compatible with bilateral pleural effusions. No lobar consolidation or pneumothorax. Mild cardiomegaly is present. The leads of a left chest wall pacer terminate in the right atrium and ventricle. The median sternotomy wires and numerous mediastinal clips are intact.

Candidate reports

CheXAgent-2: The endotracheal tube is seen with its tip residing approximately 3.8 cm above the carina. The ng tube courses into the left upper abdomen. A dual-lead pacing device is noted. Midline sternotomy wires and mediastinal clips are noted. There is diffuse pulmonary opacity which could indicate edema, ards, or pneumonia. Bilateral pleural effusions are present. The heart size is top normal. The bony structures are intact.

CheXAgent-2 + DPO (GREEN): An endotracheal tube is seen with its tip residing approximately 4.5 cm above the carina. An ng tube is seen passing into the stomach. A dual-lead pacemaker is seen with leads extending to the region of the right atrium and right ventricle. There is severe pulmonary edema with bilateral pleural effusions. The cardiac silhouette is enlarged. The mediastinal contour is stable. The bony structures are intact.

CheXAgent-2 + DPO (BERTScore): The endotracheal tube tip is approximately 5 cm from the carina. An enteric tube is seen below the diaphragm. A left-sided pacemaker device is noted with leads terminating in the right atrium and right ventricle. The heart size is mildly enlarged. The mediastinal contours are unchanged. There is moderate pulmonary edema. Small bilateral pleural effusions are present. There is no pneumothorax. No acute osseous abnormalities are seen.

CheXAgent: An endotracheal tube has been placed, which terminates in the mid thoracic trachea. An enteric tube extends inferiorly with tip in the stomach. Left sided pacer wires are in appropriate position. Sternotomy wires are intact. Small surgical clips are seen over the mediastinum. The heart is moderately enlarged. Diffuse bilateral opacities are compatible with moderate pulmonary edema. The right diaphragm is elevated with right base atelectasis. Surgical clips project over the left upper mediastinum. A small right pleural effusion is seen. There is no pneumothorax.

CheXAgent + KTO (GREEN): An endotracheal tube is seen, terminating above the carina. An enteric tube courses below the diaphragm, the tip of which is not visualized. A left sided pacemaker is seen with leads overlying the right atrium and ventricle, expected locations. The cardiac silhouette is moderately enlarged. Diffuse pulmonary opacities are present, most consistent with moderate to severe pulmonary edema. Pleural effusions may be present. No definitive pneumothorax is seen.

CheXAgent + KTO (BERTScore): An endotracheal tube is noted terminating above the carina. A left anterior chest wall pacer defibrillator is seen with leads terminating over the heart. The cardiome diastinal silhouette is enlarged. Diffuse pulmonary opacities are present, likely reflecting moderate pulmonary edema. Layering pleural effusions are also noted. There is no pneumothorax. Median sternotomy wires and mediastinal clips are noted.

Figure 8: Color-coded candidate reports: Example 3. Green and red represent correct and incorrect. Orange refers to prior imaging study. Blue indicates measurements that have not been verified.

Reviewer paper summaries: This paper introduces CheXalign, an automated pipeline for preference fine-tuning of chest X-ray (CXR) interpretation models without requiring human radiologist feedback. The authors focus on radiology report generation (RRG) for chest X-rays, a high-stakes medical domain where accuracy is crucial but obtaining expert feedback at scale is prohibitively expensive. The authors demonstrate the effectiveness of their approach by fine-tuning two models: CheXagent (8B) and CheXagent-2 (3B), achieving significant improvements in clinical efficacy metrics without additional radiologist involvement.

This paper proposes an automated pipeline for preference feedback in X-ray report generation task. Empirical analyses on the MIMIC-CXR dataset show that the performance of radiology report generation tasks can be improved using very simple preference pair generation methods. In addition, the proposed pipeline achieves state-of-the-art CheXbert scores on the MIMIC-CXR dataset in the radiology report generation task.
Summary of strengths: The paper addresses a significant challenge in medical AI - the prohibitive cost of obtaining expert radiologist feedback at scale. By developing an automated pipeline for preference pair generation using publicly available datasets and reference-based metrics, the authors provide a practical solution that could accelerate progress in medical vision-language models. In terms of the methodology and experiments, the authors conduct a thorough evaluation using multiple metrics (GREEN, LC-GREEN, BERTScore, ROUGE-L, BLEU-4, and CheXbert scores), alignment algorithms (DPO, LC-DPO, IPO, KTO, ORPO), and datasets (MIMIC-CXR and CheXpert Plus). This comprehensive assessment strengthens the validity of their findings. The proposed approach achieves notable gains, with their best setup improving CheXbert scores by 7-14% for CheXagent and attaining state-of-the-art performance on the MIMIC-CXR dataset. These improvements are consistent across different reference-based judges and assessment metrics.

* This is an early attempt that explores preference feedback in X-ray report generation task.
* The proposed pipeline is simple yet effective, achieving new state-of-the-art performance on the MIMIC-CXR dataset.
* The benchmark and evaluation results could be helpful for the research community.
Summary of weaknesses: The paper in general lacks comprehensive evaluation and analysis, and thus limits its impacts (although the authors have acknowledged some of these limitations, however, this might be necessary), this includes:
1. Lack of qualitative analysis of the model outputs to better understanding the performance.
2. Limited Model Diversity: The study focuses primarily on two models from the same family (CheXagent and CheXagent-2). It would be valuable to evaluate the approach on models with different underlying architectures to establish broader applicability.
3. Limited Analysis of Preference Pairs Quality: While the paper presents summary statistics of the generated preference pairs, there is limited analysis of their quality, consistency, or alignment with actual radiologist preferences. A more detailed characterization of the generated preference data would strengthen confidence in the approach.
4. Potential Dataset Biases: The approach relies on radiologist-written reference reports from MIMIC-CXR and CheXpert Plus. Any biases in these datasets could be amplified through preference alignment. The paper would benefit from a discussion of potential dataset biases and their implications.
5. Incomplete Analysis of Trade-offs: While the paper identifies length exploitation and introduces LC-GREEN to address it, a more comprehensive analysis of other potential reward hacking behaviors or alignment failures would provide a more complete picture of the challenges in this domain.
6. Limited Investigation of Online Alignment Algorithms: The study is restricted to offline direct alignment algorithms. As the authors note in their limitations section, exploring online or iterative alignment approaches might yield further improvements.

* The contribution could be more significant if the authors could explore new alignment algorithms more suitable for RRG tasks.
* It could be helpful to provide some human expert evaluation for the benchmark results.
Comments, suggestions and typos: I'm curious about the potential limitations of using only text-based metrics for preference pairs in a multimodal task. Since you're generating preferences based solely on comparing text reports (using GREEN or BERTScore) without considering the chest X-ray images, could this lead to misalignment between visual content and generated text?

For example, might your approach inadvertently reward reports that sound good textually but miss important visual findings? Could you share thoughts on whether this unimodal preference approach might sometimes fail to properly ground the model's outputs in the actual visual content?

Also, could you provide some examples of the preferred report and rejected report from your algorithms?

Finally, given that this is a medical application, human evaluation would provide stronger validation of the clinical utility of the improvements.

start-of-the-art -> state-of-the-art
Overall assessment scores: 3.5

4
Metareview score: 3.5
Metareview: The paper introduces a reference-based preference optimization method for x-ray medical report generation - while the results are reported on chest x-rays (using the MIMIC-CXR dataset) the method could be more general. To achieve this the authors apply the standard DAA preference learning methods, such as DPO, using on-policy preference pair rankings computed automatically using BERTScore and GREEN scores for the state-of-the-art VLMs for chest x-ray report generation: CheXAgent and CheXAgent-2. 

I consider the paper to bring forward valuable insights in the preference-tuning of chest x-ray VLMs using automated preference pair rankings. On the other hand, the methodology is pretty straightforward: there are no novel techniques or methods, the paper is just applied standard preference alignment methods from LLMs to VLMs - this was also an important suggestion from reviewers if there could be better suited preference based ranking methods that take into account the multi-modal nature of the task. 

An important weakness of the paper is the fact that the results might be overfitted on the MIMIC-CXR dataset that has been used to create the preference pairs. The results in Table 4 do not show an important alignment tax on other tasks using the same type of chest x-ray data, but I would have liked to see the results on another medical report generation dataset, for example IU-Xray. At the same time, the lack of qualitative analysis is also an important aspect the authors should improve in the camera ready version. 

Overall, the paper is of high quality and I lean for accepting it to the main conference.
Summary of reasons to publish: - Good experimental results on using preference-based alignment for medical report generation; showing state of the art on the MIMIC-CXR test set. 
- Using on-policy preference pair rankings computed automatically using BERTScore and GREEN scores for the state-of-the-art VLMs
- Highlighting the importance of having a modified GREEN score that penalizes long reports