Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?

## Abstract

Abstract Medical Vision-Language Pre-training (MedVLP) has made significant progress in enabling zero-shot tasks for medical image understanding. However, training MedVLP models typically requires large-scale datasets with paired, high-quality image-text data, which are scarce in the medical domain. Recent advancements in Large Language Models (LLMs) and diffusion models have made it possible to generate large-scale synthetic image-text pairs. This raises the question: Can MedVLP succeed using purely synthetic data? To address this, we use off-the-shelf generative models to create synthetic radiology reports and paired Chest X-ray (CXR) images, and propose an automated pipeline to build a diverse, high-quality synthetic dataset, enabling a rigorous study that isolates model and training settings, focusing entirely from the data perspective.
Our results show that MedVLP models trained exclusively on synthetic data outperform those trained on real data by 3.8% in averaged AUC on zero-shot classification. Moreover, using a combination of synthetic and real data leads to a further improvement of 9.07% . Additionally, MedVLP models trained on synthetic or mixed data consistently outperform those trained on real data in zero-shot grounding, as well as in fine-tuned classification and segmentation tasks.
Our analysis suggests MedVLP trained on well-designed synthetic data can outperform models trained on real datasets, which may be limited by low-quality samples and long-tailed distributions 1 1 1 All data and code will be released upon acceptance. .

## 1 Introduction

In medical image analysis, learning representative features typically requires labor-intensive and costly image annotations (Ronneberger et al., ; Liu et al., ). Medical Vision-Language Pre-training (MedVLP) addresses this challenge by aligning vision and language content using paired datasets of images and clinical reports, reducing the need for manual annotations (Radford et al., ; Zhang et al., ; Wu et al., ; Liu et al., ). However, existing MedVLP models rely heavily on large-scale, high-quality paired data (Liu et al., ), which is scarce in practice. Real-world datasets often contain noisy data, such as low-quality images and unpaired image-text samples, degrading model performance (Xie et al., ; Bannur et al., ).
Recent advancements in Large Language Models (LLMs) and diffusion models enable the generation of large-scale synthetic image-text datasets, offering an alternative to traditional data collection. Although these techniques have shown promise in medical tasks, they are primarily used as auxiliary support for real data via augmentation (Chen et al., ; Yao et al., ; Chen et al., ; Qin et al., ), and are often limited to single-modality settings. To the best of our knowledge, no studies have fully explored the potential of using synthetic multimodal data for MedVLP or considered training exclusively on synthetic data (Liu et al., ).

To bridge this gap and showcase synthetic data’s potential for MedVLP, our contributions are:

- •
We propose an automated pipeline to create the SynCXR dataset, which contains 200,000 synthetic images and text generated with quality and distribution control using off-the-shelf models, without relying on real data or manual curation.
- •
We successfully demonstrate that MedVLP models trained on our SynCXR dataset, containing only synthetic data, outperform those trained on real data. Moreover, combining synthetic and real data further improves performance, showcasing the effectiveness of our synthetic data generation pipeline.
- •
We identify several issues in the most commonly used real dataset for MedVLP, MIMIC-CXR (Johnson et al., ), that degrade MedVLP performance, including low-quality images and unpaired image-text samples. Furthermore, we identify the long-tailed distribution problem in multimodal datasets, as shown in Fig 1, 2.
- •
We conduct an extensive analysis of the key factors contributing to MedVLP’s success using purely synthetic data. Our method is evaluated on seven downstream tasks using zero-shot learning and linear probing, demonstrating that MedVLP can effectively perform with synthetic data alone.

## 2 Related Work

Figure: Figure 1: Comparison of real image-text datasets and synthetic datasets. (a): The real image-text dataset, MIMIC-CXR (Johnson et al., ), while authentic, often contains imperfections such as long-tailed data distribution, unpaired images and text, and low-quality CXR images, which limit the performance of MedVLP models pretrained on this dataset. (b): The synthetic dataset generation process uses clinical entities as prompts to an LLM (e.g., Llama3.1 (AI@Meta, )) to generate synthetic reports. These reports are then used to create synthetic images through RoentGen (Bluethgen et al., ). We propose an automated pipeline to control the dataset distribution, ensuring it is balanced and includes paired image-text samples.
Refer to caption: extracted/5934507/fig1.png

Representation Learning with Synthetic Data.
Synthetic data has been widely employed across various deep learning fields (Rossenbach et al., ; Varol et al., ; Jahanian et al., ; Zhou et al., ; Yang et al., ; Li et al., ). In visual representation learning, synthetic data has improved model performance in a range of tasks (Richter et al., ; Ros et al., ; Chen et al., ; Johnson-Roberson et al., ; Yuan et al., ; Shmelkov et al., ). Recent efforts have also focused on using synthetic data from text-to-image models to augment real-world data during training (Azizi et al., ; Sariyildiz et al., ; He et al., ). For example, (Yu et al., ) introduced a framework to generate synthetic images to diversify existing datasets. Notably, methods utilizing text-to-image generative models (Rombach et al., ) have demonstrated that synthetic images guided by real captions can effectively train self-supervised models, achieving performance comparable to that of real images (Tian et al., ).

Further advancements like SynCLR (Tian et al., ) have focused on visual representation learning using only synthetic images, generated with conditioning on various categories. Meanwhile, other recent works (Fan et al., ; Sharifzadeh et al., ; Xie et al., ) have explored joint image and text generation for enhanced vision-language pretraining (VLP). However, only one study, SynthCLIP (Hammoud et al., ), investigates VLP exclusively with synthetic data, and even that work is limited to natural images. To date, no research has explored the potential of MedVLP trained solely on synthetic data.

Medical Vision Language Pre-training.
Recent work on MedVLP has focused on integrating visual and textual modalities, particularly for chest X-ray (CXR) images. Studies such as (Zhang et al., ; Huang et al., ; Wang et al., ; Liu et al., ; ; ; Wan et al., ) have concentrated on aligning CXR images with paired radiology reports. Some methods also leverage external datasets to boost performance, raising concerns about generalizability (Wu et al., ; Zhang et al., ; Li et al., ; Phan et al., ).
However, all current MedVLP approaches rely heavily on large-scale, real image-text paired datasets like MIMIC-CXR (Johnson et al., ). Some even require additional human-annotated datasets or manual interventions to improve model performance (Wu et al., ; Zhang et al., ; Phan et al., ), which limits their scalability and accessibility.

Synthetic Data for Medical Image Tasks.
Given the scarcity of annotated data, high costs, and privacy concerns in medical data collection, synthetic data has been explored to support various medical image tasks (Koetzier et al., ). However, most prior work focuses on image modality and supervised learning (Chen et al., ; Yao et al., ; Chen et al., ; Qin et al., ), using synthetic data solely as augmentation for real datasets (Khosravi et al., ; Ktena et al., ). Few studies have evaluated models trained entirely on synthetic medical data (Wu et al., ). Recent efforts have generated synthetic text and images for MedVLP (Xie et al., ), but still restrict synthetic data usage to augmentation. Consequently, the full potential of synthetic data in MedVLP remains largely unexplored.

In this work, we generate both synthetic CXR images and reports, then training a MedVLP model solely on synthetic data. We conduct an extensive evaluation of the impact of large-scale synthetic medical data on MedVLP, exploring its performance across various downstream tasks.

## 3 Methods

Figure: Figure 2: (a): Examples of invalid or low-quality images filtered out by the proposed image curation method described in Sec 3.1. (b): The image curation pipeline uses InternVL2 (Chen et al., ), a Multimodal Large Language Model (MLLM), to assess CXR image quality. Images that meet the criteria are retained; others are discarded. (c): Entity frequency distribution in the MIMIC-CXR dataset. Due to space constraints, only the top 50 frequent entities for four categories (Abnormality, Non-Abnormality, Disease, Non-Disease) are shown. A more detailed distribution is presented in Fig 7,8,11,9,10.
Refer to caption: extracted/5934507/badsample.png

### 3.1 Exploring Imperfections in Real Data

For MedVLP, the most commonly used dataset is MIMIC-CXR (Johnson et al., ; ), a collection of chest x-ray (CXR) images paired with their corresponding textual reports. after following the preprocessing steps outlined in previous works (Zhang et al., ; Wang et al., ; Huang et al., ), this dataset provides a total of 213,384 image-text pairs for pre-training. And all images must be frontal views according to the preprocessing steps outlined in (Huang et al., ).

Previous work on VLP with natural images (Xu et al., ) has shown that data quality, including image fidelity and long-tailed distribution, significantly impacts model performance. However, the quality of MedVLP datasets remains underexplored due to ambiguity in defining medical image quality, stemming from diverse imaging protocols. Additionally, quantifying data distribution is complex, as radiology reports often describe patterns across multiple anatomical regions rather than distinct categories.To address these challenges, we develop a systematic pipeline to thoroughly analyze the data issues in the MIMIC-CXR (Johnson et al., ) dataset, as detailed in the following sections.

Low-Quality and Mismatched Image-Text Pairs.
Our aim is to explore and identify issues related to image quality in the MIMIC-CXR dataset (Johnson et al., ), rather than to completely clean the dataset, as creating a perfect dataset and filtering out all low-quality samples is infeasible for large-scale multimodal datasets (Xu et al., ).
Inspired by (Bannur et al., ), which highlights various issues with poor-quality images, we design six queries for a Multimodal Large Language Model (MLLM), utilizing the InternVL2-26B model(^2^22https://huggingface.co/OpenGVLab/InternVL2-26B) (Chen et al., ; ). Each CXR image from the MIMIC-CXR dataset is paired with these six queries, and the MLLM process each query independently. The process is depicted in Fig 2 (b).

- •
Detecting Non-CXR Images: <CXR Image>, Please check if the given image is a chest X-ray scan. If it is a chest X-ray, return ‘YES’. Otherwise, return ‘NO’.
- •
Detecting Non-Human CXR Images: <CXR Image>, Please verify if the given image is a human chest X-ray scan. If it is a chest X-ray, return ‘YES’. Otherwise, return ‘NO’.
- •
Detecting Wrong Views: <CXR Image>, Please check if the given image is a frontal chest X-ray view. If it is a frontal view, return ‘YES’. If it is a lateral view or any other view, return ‘NO’.
- •
Assessing Image Quality: <CXR Image>, Please analyze the provided chest X-ray (CXR) image and respond with ‘NO’ if the image quality is poor, such as being blurry, containing artifacts, or having poor contrast. Respond with ‘YES’ if the image quality is acceptable.
- •
Detecting Artifacts and Overprocessing: <CXR Image>, Please analyze the following chest X-ray image. Respond with ‘YES’ if the image is clear, correctly oriented, and free of artifacts or imperfections that could affect its diagnostic quality. Respond with ‘NO’ if the image is blurry, incorrectly oriented, contains artifacts, or has imperfections that make it unsuitable for further analysis.
- •
Checking High-Fidelity: <CXR Image>, Please check if the given image is a high-fidelity human chest X-ray scan. If it is a high-fidelity chest X-ray, return ‘YES’. Otherwise, return ‘NO’.

After this process, we filter out the CXR images where the answers are all ‘NO’ across the six queries. Fig 2 (a) shows examples of images where the answer was ‘NO’. We identified and removed 1,448 such images and their corresponding reports from the preprocessed MIMIC-CXR dataset, leaving us with 211,936 image-text pairs.

To further refine the dataset, we use the CXR-specific vision encoder, RAD-DINO (Pérez-García et al., ), to extract image features from the remaining 211,936 CXR images and from the 1,448 samples identified as bad by MLLM filtering. We then compute the similarity between each image in the cleaned dataset and each of the bad samples. Since each image comes from a different clinical case, we only compare image quality rather than the clinical content (e.g., diagnoses or abnormalities). To do this, we set a similarity threshold of 0.5 and remove all images with a similarity score greater than 0.5. This step resulted in the removal of an additional 5,512 images and their paired reports, reducing the dataset to 206,424 image-text pairs.
Fig 2 (a) also shows the samples removed based on their similarity to bad images using visual features from RAD-DINO(^3^33https://huggingface.co/microsoft/rad-dino) (Pérez-García et al., ).

In our exploration of the MIMIC-CXR dataset, we utilized a rough approach to identify problematic images, such as non-chest images, wrong views, overprocessing, and low-fidelity scans. Our results confirm that many images in the dataset exhibit these issues. While our approach identifies numerous problematic images, fully curating and removing all low-quality cases is unfeasible due to the substantial human effort required and the absence of well-defined criteria for an automated cleaning pipeline.
Furthermore, addressing all instances of low-quality images remains highly challenging through automated processes alone.

Uncovering Long-tailed Distribution in MIMIC-CXR.
As demonstrated in previous work on natural image-text data (Xu et al., ; Hammoud et al., ), a long-tailed distribution in VLP datasets negatively impacts model performance.
Therefore, we aim to explore the data distribution of the MIMIC-CXR dataset. However, directly evaluating the text distribution at the sample level, as done in (Xu et al., ), is challenging because each radiology report often describes multiple patterns or anatomical regions, unlike natural image captions that typically focus on a single object (Zhang et al., ).

Instead, we adopt an alternative approach by using an off-the-shelf Named Entity Recognition (NER) tool to extract all medical entities, treating them as representatives of the report’s concepts and exploring the dataset distribution at the entity level. For this, we use RaTE(^4^44https://huggingface.co/Angelakeke/RaTE-NER-Deberta) (Zhao et al., ), a model specifically designed for NER tasks on radiology reports.
RaTE automatically classifies the extracted entities into five categories: [ABNORMALITY, NON-ABNORMALITY, DISEASE, NON-DISEASE, ANATOMY].
We display the top 50 frequent entiites distribution of each entity type in Fig 2 (c).
We display the top 50 frequent entiites distribution of each entity type in Fig 7,8,11,9,10. As shown, all entity types exhibit a severe long-tailed distribution. The MIMIC-CXR (Johnson et al., ) includes a total of 154,049 unique entities, with 55,047 Abnormality, 36,365 Non-Abnormality, 23,017 Disease, 22,103 Non-Disease, and 40,517 Anatomy entities.

### 3.2 Generating Synthetic CXR reports and Paired Images.

Since the MIMIC-CXR dataset (Johnson et al., ) contains various data issues, we generate synthetic radiology reports and CXR images, controlling data quality and distribution during generation to alleviate these problems.
In this work, we aim to explore the effectiveness of pretraining MedVLP on a purely synthetic dataset, rather than attempting to create a perfect dataset, as noisy data is unavoidable in real-world scenarios and an ideal dataset is unrealistic.

CXR Report Generation.
To generate the synthetic reports, the pipeline is depicted in Fig 6. We select a general LLM, Llama3.1-70B-Instruct(^5^55https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) as the report generator, and we extensively ablate the performance of the report generator with other LLMs in Fig 4. We query the LLM using prompts that include the entity list, as shown in Fig 6.

Since we aim to build a synthetic dataset without a long-tailed distribution, we design a balanced sampling strategy to ensure that the appearance frequency of each entity type is approximately equal across the synthetic dataset. Let $\mathcal{E}$ be the set of entities, categorized into five types: ABNORMALITY, NON-ABNORMALITY, DISEASE, NON-DISEASE, and ANATOMY.

For each generation, we sample:

$$ $\mathcal{S}_{1}=\{e_{1}^{(i)},e_{2}^{(i)},\ldots,e_{k}^{(i)}\},\quad\forall e_ {j}^{(i)}\in\{\texttt{{ABNORMALITY}},\texttt{{NON-ABNORMALITY}},\texttt{{ DISEASE}},\texttt{{NON-DISEASE}}\}$ $$

where $k$ is the number of entities sampled from the first four categories. Additionally, we sample:

$$ $\mathcal{S}_{2}=\{a_{1}^{(i)},a_{2}^{(i)},\ldots,a_{m}^{(i)}\},\quad\forall a_ {j}^{(i)}\in\texttt{{ANATOMY}}$ $$

where $m$ is the number of entities sampled from the ANATOMY category. Thus, the total sampled entity set for each generation is:

$$ $\mathcal{S}=\mathcal{S}_{1}\cup\mathcal{S}_{2}$ $$

We impose a maximum frequency threshold, $\tau_{\max}$, for each entity $e\in\mathcal{E}$. If an entity $e_{j}^{(i)}$ in $\mathcal{S}$ reaches this threshold, we resample $e_{j}^{(i)}$ while keeping the remaining entities in $\mathcal{S}$ unchanged:

$$ $\text{if }f(e_{j}^{(i)})\geq\tau_{\max},\text{ then resample }e_{j}^{(i)}.$ $$

Here, $f(e)$ denotes the current frequency of entity $e$ in the dataset. This ensures a balanced distribution of entities across the synthetic dataset.

After sampling, we input the selected entities $\mathcal{S}=\mathcal{S}_{1}\cup\mathcal{S}_{2}$ into the LLM and indicate their type. Let the output of the LLM be denoted as $R_{gen}$, which represents the synthetic report generated by the model based on the sampled entities. To ensure that the LLM-generated report $R_{gen}$ covers and only includes the entities in $\mathcal{S}$ (since the inclusion of non-specified entities would disrupt the frequency balance), we use the RaTE model (Zhao et al., ) to extract entities from $R_{gen}$, denoted as $\mathcal{E}_{gen}$.

We then verify the entity set $\mathcal{E}_{gen}$ by comparing it with the originally sampled set $\mathcal{S}$. If $\mathcal{E}_{gen}\neq\mathcal{S}$, we regenerate the report $R_{gen}$ by repeating the generation process until $\mathcal{E}_{gen}=\mathcal{S}$:

$$ $\text{if }\mathcal{E}_{gen}\neq\mathcal{S},\text{ regenerate }R_{gen}\text{ until }\mathcal{E}_{gen}=\mathcal{S}.$ $$

Once the synthetic report is successfully generated, it is used as the ‘FINDINGS’ section of the CXR report. We then query the LLM to summarize $R_{gen}$ into the ‘IMPRESSION’ section, denoted as $R_{imp}$. To ensure consistency between the entities in the ‘FINDINGS’ and ‘IMPRESSION’ sections, we extract entities from the summary $R_{imp}$ using RaTE, denoted as $\mathcal{E}_{imp}$. We verify that:

$$ $\mathcal{E}_{imp}=\mathcal{S}.$ $$

If the entities in $R_{imp}$ do not match $\mathcal{S}$, we regenerate the "IMPRESSION" section until $\mathcal{E}_{imp}=\mathcal{S}$:

$$ $\text{if }\mathcal{E}_{imp}\neq\mathcal{S},\text{ regenerate }R_{imp}\text{ until }\mathcal{E}_{imp}=\mathcal{S}.$ $$

Given that the number of samples in the original MIMIC-CXR dataset cannot be perfectly divided by $k$ and $m$, we generate a total of 200,000 synthetic samples to ensure a balanced distribution using only off-the-shelf tools, without any specific design for CXR data.

While RadGraph (Delbrouck et al., ) could be used for entity extraction, it relies on human-annotated data from MIMIC-CXR and is limited to 16,117 entities. In contrast, RaTE (Zhao et al., ) extracts 154,049 entities, making it more suitable for our goal of creating a general and easily transferable pipeline for synthetic data generation. Thus, we chose RaTE for its broader applicability to various radiology reports.

CXR Image Generation.
After generating the synthetic radiology reports, we aim to generate paired CXR images conditioned on the synthetic reports. Since general text-to-image (T2I) models (e.g., Stable Diffusion) are not designed for CXR image generation and demonstrate poor performance, as shown in (Liu et al., ; Bluethgen et al., ), we select RoentGen(^6^66https://stanfordmimi.github.io/RoentGen/) (Bluethgen et al., ), the most recent and validated CXR-specific T2I model, verified by clinicians, as our image generator.
We use RoentGen’s (Bluethgen et al., ) official pretrained weights to generate images. Following their implementation, we use only the ‘IMPRESSION’ section from the synthetic reports as the text prompt for the T2I model. The generation process is controlled using the official hyperparameters provided by RoentGen, where the classifier-free guidance (CFG) is set to 4 and the number of denoising steps is set to 50.

To prevent the synthetic images from exhibiting the same issues found in the real dataset (as discussed in Sec. 3.1), we apply a similar curation procedure. First, we use the MLLM to filter synthetic images, and then we compute the similarity of visual features between synthetic images and the problematic samples identified from the real dataset. If the visual similarity exceeds a threshold $\delta=0.5$, we regenerate the images by re-querying the T2I model with the same text prompt until they pass the curation procedure.

We generate 200,000 synthetic CXR images, each paired with a corresponding synthetic report, using only general-purpose, open-source models (e.g., Llama3.1 (AI@Meta, ), InternVL2 (Chen et al., )) and vision models pre-trained with self-supervised learning (e.g., RAD-DINO (Pérez-García et al., )). No annotated CXR images or MedVLP models pre-trained on specific CXR image-text datasets are used in this process. This ensures our approach is adaptable and can easily incorporate future advancements in general-purpose models. We refer to this dataset as SynCXR.

### 3.3 Synthetic Data Training for MedVLP

Finally, we use the synthetic dataset, SynCXR, to train a MedVLP model and explore how effectively a model can learn from pure synthetic data. Since there are many existing methods for MedVLP, we select simple baseline models like ConVIRT (Zhang et al., ) and GLoRIA (Huang et al., ) for the following reasons:

ConVIRT (Zhang et al., ) jointly trains vision and text encoders on paired medical images and reports using global contrastive learning.

GLoRIA (Huang et al., ) extends ConVIRT by incorporating both global and regional contrastive learning to train the encoders on paired medical images and reports.

These models are open-source, straightforward, and minimize the influence of external factors on evaluating synthetic data for MedVLP. For retraining these two methods on our synthetic dataset, SynCXR, we strictly use their official codebases(^7^77https://github.com/marshuang80/gloria)(^8^88https://github.com/edreisMD/ConVIRT-pytorch). More complex models may introduce unnecessary complications.

Excluding Complex Models.
Recent models like BioViL (Boecking et al., ) and BioViL-T (Bannur et al., ) lack publicly available training code, making them impractical for re-training with synthetic data. Knowledge-enhanced MedVLP models such as MedKLIP, KAD, and MAVL (Wu et al., ; Zhang et al., ; Phan et al., ) rely on external tools and human-annotated data to incorporate additional knowledge, making direct implementation with synthetic data challenging and introducing unnecessary variables.

## 4 Experiments Configurations

For pre-training, we apply the official configurations provided by ConVIRT (Zhang et al., ) and GLoRIA (Huang et al., ) on the MIMIC-CXR dataset to our synthetic CXR image-text dataset, SynCXR.

### 4.1 Downstream Task Datasets and Configurations

For downstream tasks, we evaluate the effectiveness of synthetic data for MedVLP across four tasks. Details on the datasets and implementation are provided in Appendix, Sec A.

Zero-shot Medical Image Classification.
Following the guidelines in (Phan et al., ; Wu et al., ), we perform this task on seven datasets: CheXpert (Saporta et al., ), ChestXray-14 (Wang et al., ), PadChest-seen, PadChest-unseen, PadChest-rare (Bustos et al., ), RSNA (Shih et al., ), and SIIM (Steven G. Langer & George Shih, ), using the dataset splits from (Phan et al., ). Evaluation metrics include AUC, F1, and ACC.

Zero-shot Medical Image Visual Grounding.
In line with (Phan et al., ), this task is conducted on the RSNA (Shih et al., ), SIIM (Steven G. Langer & George Shih, ), and Covid-19 Rural (Desai et al., ) datasets, using official splits and metrics. Grounding performance is evaluated with IoU, and Dice score.

Medical Image Fine-tuned Classification.
As described in (Phan et al., ), we use the RSNA (Shih et al., ), SIIM (Steven G. Langer & George Shih, ), Covid-19 CXR-2 (Pavlova et al., ), and ChestXray-14 (Wang et al., ) datasets. During fine-tuning, all model parameters, including the pre-trained vision encoder and linear classifier, are updated. The AdamW optimizer is applied with a learning rate of $1\times 10^{-4}$, batch size of 64, and training runs for 50 epochs. Evaluation follows the AUC score protocol in (Huang et al., ; Wang et al., ; ).

Medical Image Fine-tuned Segmentation.
This task uses the RSNA (Shih et al., ), SIIM (Steven G. Langer & George Shih, ), and Covid-19 Rural (Desai et al., ) datasets, following preprocessing from (Wang et al., ; Huang et al., ). U-Net (Ronneberger et al., ) is used for fine-tuning, freezing the pre-trained vision encoder and updating only the decoder parameters. Performance is measured using the Dice score, adhering to the evaluation protocol from (Huang et al., ).

### 4.2 Experimental Results

Since the MIMIC-CXR dataset already includes several diseases present in downstream tasks, as mentioned in (Phan et al., ; Zhang et al., ), we split the zero-shot classification task into seen and unseen categories, strictly following (Phan et al., ). Note that all experimental results for ConVIRT and GLoRIA pre-trained with real data (MIMIC-CXR) are directly referenced from (Phan et al., ) to ensure a fair comparison.

Zero-shot Classification on Seen Diseases.
Tab 1 shows the zero-shot classification performance on seen diseases. Across all datasets, both MedVLP methods pretrained on SynCXR (our purely synthetic dataset) consistently outperform or achieve comparable performance to their counterparts pretrained on real datasets, with an average improvement of 4.7% in AUC and 4.53% in F1 scores. Furthermore, the methods pretrained on the mixed dataset, which directly combines real and synthetic data, achieve even greater improvements, with 10.08% AUC and 7.62% F1 scores on average across all datasets and methods. This demonstrates that the SynCXR dataset effectively enables MedVLP models to learn representative cross-modal features, enhancing their zero-shot classification capability.

Zero-shot Classification on Unseen Diseases.
Tab 3(a) reports the zero-shot classification performance on unseen diseases. Similar to the results for seen diseases, MedVLP models pretrained on the synthetic dataset consistently outperform those pretrained on real data, with an average improvement of 2.96% AUC and 0.51% F1 scores. Additionally, models pretrained on the mixed dataset show substantial gains over those trained on real data, with 7.39% AUC and 1.52% F1 scores on average. This indicates that the SynCXR dataset, generated with meticulous quality control and balanced distribution, can increase the generalizability of MedVLP models for unseen diseases prediction.

Zero-shot Visual Grounding.
We further evaluate the effectiveness of synthetic data in improving MedVLP models’ local visual understanding capabilities through zero-shot grounding tasks. Tab 3(b) presents the performance of zero-shot grounding on RSNA (Shih et al., ), Covid-19 Rural (Desai et al., ), and SIIM (Steven G. Langer & George Shih, ). Across all datasets, MedVLP models pretrained on the SynCXR dataset achieve superior performance compared to those trained on the real dataset, with an average increase of 1.42% IoU and 0.97% Dice scores. The mixed dataset further enhances performance, with 4.06% IoU and 2.92% Dice scores on average. This demonstrates that the SynCXR dataset not only benefits global cross-modal feature learning but also improves local visual understanding for MedVLP models.

**Table 1: Performance of zero-shot classification on five datasets for diseases present in the MIMIC-CXR dataset, evaluated on two MedVLP models pretrained on MIMIC-CXR (real) and SynCXR (pure synthetic). ‘Mix’ denotes the direct combination of real and synthetic data for MedVLP pretraining. Best results are highlighted in bold.**
| Method | Pre-training | CheXpert | ChestXray-14 | PadChest-seen | RSNA | SIIM |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Data | AUC $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | F1 $\uparrow$ |
| ConVIRT | MIMIC-CXR | 52.10 | 35.61 | 53.15 | 12.38 | 63.72 | 14.56 | 79.21 | 55.67 | 64.25 | 42.87 |
| SynCXR | 59.49 | 40.51 | 56.07 | 15.43 | 63.43 | 15.10 | 82.08 | 58.38 | 75.55 | 57.43 |  |
| Mix | 71.54 | 47.11 | 61.28 | 18.52 | 68.48 | 16.67 | 83.86 | 61.28 | 78.51 | 59.10 |  |
| GLoRIA | MIMIC-CXR | 54.84 | 37.86 | 55.92 | 14.20 | 64.09 | 14.83 | 70.37 | 48.19 | 54.71 | 40.39 |
| SynCXR | 61.38 | 41.05 | 57.47 | 15.60 | 64.26 | 15.02 | 72.34 | 49.50 | 67.32 | 53.86 |  |
| Mix | 72.32 | 48.54 | 61.06 | 17.33 | 68.35 | 17.00 | 74.32 | 51.10 | 73.49 | 56.09 |  |

**(a)**
| Method | Pre-training | Covid-19 CXR-2 | PadChest-unseen | PadChest-rare |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Data | AUC $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | F1 $\uparrow$ |
| ConVIRT | MIMIC-CXR | 62.78 | 71.23 | 51.17 | 4.12 | 50.37 | 3.31 |
| SynCXR | 64.41 | 72.03 | 54.47 | 4.51 | 53.70 | 3.69 |  |
| Mix | 69.23 | 72.85 | 58.53 | 5.35 | 57.68 | 4.40 |  |
| GLoRIA | MIMIC-CXR | 64.52 | 70.78 | 49.96 | 4.07 | 48.25 | 3.41 |
| SynCXR | 66.70 | 71.90 | 54.24 | 4.10 | 51.26 | 3.75 |  |
| Mix | 68.76 | 73.22 | 58.60 | 5.60 | 58.58 | 4.62 |  |

Fine-tuning Tasks.
To evaluate the representation quality learned by MedVLP, we report the fine-tuned classification and segmentation performance in Tab 3. Similar to the zero-shot task, MedVLP models pre-trained on SynCXR consistently outperform those trained on the real dataset across all data ratios for both classification and segmentation tasks. Furthermore, the combination of real and synthetic datasets (Mix) further boosts performance, demonstrating that SynCXR data not only enhances cross-modal representation learning but also improves performance in single-modal tasks.

**Table 3: Results from two MedVLP methods pre-trained on real, synthetic, and mixed datasets are reported for classification (AUC) and segmentation (Dice) tasks. ‘ConVIRT-Real’ and ‘GLoRIA-Real’ refer to models pre-trained on MIMIC-CXR using real data, while ‘ConVIRT-Syn’ and ‘GLoRIA-Syn’ indicate models pre-trained on SynCXR using synthetic data. ‘ConVIRT-Mix’ and ‘GLoRIA-Mix’ represent models trained on a combination of MIMIC-CXR and SynCXR. Best results are in bold.**
| Task | Classification | Segmentation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | RSNA | SIIM | Covid19 CXR-2 | ChestXray-14 | RSNA | Covid-19 Rural | SIIM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Data Ratio | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% |
| ConVIRT-Real | 78.86 | 85.42 | 87.64 | 72.39 | 80.41 | 91.67 | 90.30 | 97.74 | 99.70 | 57.23 | 72.53 | 79.13 | 56.48 | 63.94 | 71.87 | 16.97 | 30.79 | 42.71 | 28.75 | 47.21 | 65.75 |
| ConVIRT-Syn | 79.01 | 85.58 | 87.90 | 73.51 | 81.10 | 91.84 | 91.50 | 98.80 | 99.73 | 57.45 | 73.60 | 80.20 | 58.00 | 65.10 | 72.90 | 17.10 | 32.00 | 43.90 | 29.90 | 48.50 | 66.81 |
| ConVIRT-Mix | 79.75 | 86.21 | 88.45 | 73.00 | 82.80 | 92.31 | 91.81 | 99.00 | 99.81 | 57.61 | 74.20 | 80.51 | 58.50 | 65.81 | 73.30 | 18.40 | 32.50 | 44.21 | 30.10 | 48.81 | 67.11 |
| GLoRIA-Real | 79.13 | 85.59 | 87.83 | 75.85 | 86.20 | 91.89 | 92.74 | 97.18 | 99.54 | 58.94 | 72.87 | 79.92 | 58.13 | 67.71 | 72.06 | 16.12 | 31.20 | 43.85 | 31.87 | 40.61 | 64.82 |
| GLoRIA-Syn | 80.30 | 86.75 | 88.00 | 76.01 | 87.40 | 92.11 | 94.01 | 98.41 | 99.75 | 60.11 | 74.01 | 81.11 | 60.41 | 70.01 | 73.51 | 17.31 | 32.51 | 45.01 | 32.91 | 41.91 | 66.01 |
| GLoRIA-Mix | 81.01 | 87.50 | 88.61 | 77.51 | 88.01 | 92.51 | 94.51 | 99.61 | 99.86 | 60.31 | 74.51 | 81.51 | 61.01 | 70.51 | 74.01 | 17.51 | 33.01 | 45.31 | 33.51 | 42.21 | 67.51 |

## 5 Analysis

**(a)**
| Method | Entity Sampling | Avg. Zero-shot |
| --- | --- | --- |
| Strategy | Classification |  |
| ConVIRT (Zhang et al., ) | w/ balance Sampling | 63.65 |
| w/o balance Sampling | 60.21 |  |
| GLoRIA (Huang et al., ) | w/ balance Sampling | 61.87 |
| w/o balance Sampling | 58.42 |  |

Effect of Balanced Entity Sampling in Generating Synthetic Reports.
We evaluate the impact of balanced sampling entities when generating synthetic reports using LLMs. For the synthetic dataset without balanced sampling, we adjust entity frequencies to match their distribution in MIMIC-CXR, leading to a long-tailed distribution. As shown in Tab 4(a), for both MedVLP methods, the performance improves significantly when using synthetic datasets generated from balanced sampled entities. This demonstrates that balanced sampling of entities leads to a more representative dataset, benefiting MedVLP performance.

Evaluating the Contribution of Synthetic Images and Reports.
We aim to assess the individual impact of synthetic images and synthetic reports on MedVLP performance. As shown in Tab 4(b), we generate two partially synthetic datasets by replacing either the image or the text with synthetic data, while keeping the other components real, to evaluate their respective contributions.

- •
Real Image, Synthetic Report: In this setting, we use MedVersa(^9^99[https://huggingface.co/hyzhou/MedVersa](https://huggingface.co/hyzhou/MedVersa)) (Zhou et al., ), a state-of-the-art radiology report generation model, to generate synthetic reports for each real CXR image. We then train MedVLP models using these real image and synthetic report pairs.
- •
Real Report, Synthetic Image: In this setting, we use RoentGen (Bluethgen et al., ), a text-to-image model, to generate synthetic CXR images for each real report. The ‘IMPRESSION’ section of each report serves as the prompt for generating synthetic CXR images. These synthetic image and real report pairs are used to train MedVLP models.

According to Tab 4(b), for both MedVLP methods, using real images with synthetic reports results in decreased performance, likely due to the persistent long-tailed distribution, as the synthetic reports are generated based on real images. However, using real reports with synthetic images slightly improves performance, as synthetic images can be curated using our image filtering procedure to ensure high quality, avoiding issues commonly found in real datasets. Using both synthetic images and synthetic reports achieves the highest performance, indicating that a well-curated synthetic dataset can significantly enhance MedVLP performance.

Figure: Figure 4: Effectiveness of various factors on SynCXR dataset. Top: Impact of entity usage ratio on MedVLP performance for ConVIRT and GLoRIA methods. Bottom Left: Effectiveness of different LLMs for report generation on both MedVLP methods. Bottom Right: Effectiveness of different CXR image generation models for both MedVLP methods.
Refer to caption: extracted/5934507/ablate.png

Impact of Entity Diversity.
We evaluate the impact of entity diversity by varying the number of entities used for generating the SynCXR dataset. We generate synthetic datasets using 25%, 50%, and 75% of these entities, following the same procedure each time. The results, shown in Fig 4 (Top), indicate that zero-shot classification performance improves as more entities are used for report generation. This suggests that increasing dataset diversity positively influences downstream performance.

Impact of Different Report Generators.
We also examine the impact of using different LLMs for synthetic report generation. As shown in Fig 4 (Bottom Left), we compare two general LLMs, LLaMA 3.1 (8B and 70B), and two medical-specific LLMs, Meditron3 (8B(^10^1010[https://huggingface.co/OpenMeditron/Meditron3-8B](https://huggingface.co/OpenMeditron/Meditron3-8B)) and 70B(^11^1111[https://huggingface.co/OpenMeditron/Meditron3-70B](https://huggingface.co/OpenMeditron/Meditron3-70B))). Despite Meditron3 being trained specifically on medical corpora and inheriting weights from LLaMA, the dataset generated by LLaMA 3.1-70B-Instruct achieves the best performance. This indicates that a powerful general LLM is effective for generating synthetic datasets, and using domain-specific fine-tuned versions may degrade the quality of the synthetic data.

Impact of Different Image Generators.
We evaluate various text-to-image models for synthetic CXR image generation, including CXR-IRGen (Shentu & Al Moubayed, ), LLM-CXR (Lee et al., ), and RoentGen (Bluethgen et al., ). As shown in Fig 4 (Bottom Right), datasets generated by RoentGen lead to the best performance for both MedVLP methods. This is likely because RoentGen is the only image generation model verified by clinicians, suggesting that the quality of image generation models is crucial for building synthetic datasets, and models should be validated by clinical experts.

## 6 Conclusion

In this work, we tackle the question: Can MedVLP succeed using purely synthetic data? Our findings demonstrate that the answer is: Yes. To the best of our knowledge, this is the first study to comprehensively explore the potential of synthetic data for MedVLP models. We also identify key limitations in existing real-world datasets and introduce SynCXR—a synthetic dataset of 200,000 image-text pairs generated without any manual quality checks.
Our findings show that MedVLP models trained on purely synthetic data outperform those trained on real data. Moreover, combining synthetic and real data further boosts model performance, demonstrating the potential of synthetic data to overcome limitations in real-world datasets. We systematically analyze key factors in SynCXR and validate its effectiveness through extensive ablation studies.
In summary, we show that MedVLP achieves strong performance using a purely synthetic image-text dataset and benefits significantly from a combination of real and synthetic data. We believe this work will inspire the community to fully leverage synthetic data and mitigate the challenges posed by noisy and limited real-world datasets.

## Appendix A Downstream Tasks Configuration

### A.1 Dataset Details

In this section, we provide details on all datasets used. The dataset splits are publicly accessible at(^12^1212https://github.com/HieuPhan33/CVPR2024_MAVL/tree/main/data).

ChestX-ray14 (Wang et al., ) includes 112,120 frontal X-rays from 30,805 patients, labeled for 14 diseases. We use the official split and partition it into 80%/10%/10% for train/validation/test.

PadChest (Bustos et al., ) includes 160,868 X-rays from 67,000 patients, annotated with over 150 findings. As in (Phan et al., ), three subsets are built based on PadChest: 14 common diseases as PadChest-seen, rare diseases from the NORD database(^13^1313https://rarediseases.org/rare-diseases/) as PadChest-rare, and the remaining diseases as PadChest-unseen. We use the official split provided by (Phan et al., ).

RSNA (Shih et al., ) contains over 260,000 frontal X-rays annotated with pneumonia masks. We divide it into training (60%), validation (20%), and test (20%) sets for segmentation and classification tasks (Huang et al., ; Wu et al., ).

CheXpert (Irvin et al., ) contains 224,316 chest X-rays from 65,240 patients at Stanford Hospital, with an official validation set of 200 studies and a test set of 500 studies, both annotated by board-certified radiologists. Our evaluation on the five observations in the official test set follows protocols from earlier studies (Tiu et al., ; Irvin et al., ).

SIIM (Steven G. Langer & George Shih, ) consists of over 12,000 frontal X-rays annotated with pneumothorax masks, split into training (60%), validation (20%), and test (20%) sets.

COVIDx CXR-2 (Wang et al., ) includes 29,986 X-rays from 16,648 COVID-19 patients, divided into training (70%), validation (20%), and test (10%) (Pavlova et al., ).

COVID Rural (Desai et al., ) contains over 200 X-rays with segmentation masks, divided into training (60%), validation (20%), and test (20%).

### A.2 Implementation Details

Zero-shot Image Classification.
The CXR images undergo a two-step preprocessing: resizing to $256\times 256$, followed by center cropping to $224\times 224$. As per (Huang et al., ), pixel values are normalized to $[0,1]$. The processed image is passed through a visual encoder and projector to generate the image embedding $\hat{\mathbf{v}}_{i}$. Simultaneously, the text prompts are processed through a text encoder to obtain text embeddings $\hat{\mathbf{l}}_{i}$. Classification is based on cosine similarity between image and text embeddings. If the similarity between the image embedding and the positive prompt (e.g., disease) is higher than that with the negative prompt (e.g., No disease), the classification is positive, and vice versa. The prompt design follows (Tiu et al., ) for both ConVIRT and GLoRIA.

Zero-shot Visual Grounding.
For this task, we follow the BioViL pipeline as described in (Phan et al., ), since ConVIRT (Zhang et al., ) and GLoRIA (Huang et al., ) do not provide code for visual grounding. This pixel-level classification task relies on the similarity between text embeddings and the dense visual feature map from the final convolutional layer. The cosine similarity generates a similarity map, resized to match the image, and used as segmentation results for grounding evaluation.

Medical Image Fine-tuned Classification.

For fine-tuning, we follow the experimental setup from (Phan et al., ), updating both the visual encoder and linear layer. Images are resized to $256\times 256$, and data augmentation is applied as recommended in (Zhang et al., ). We use the AdamW optimizer with a learning rate of $1\times 10^{-4}$, batch size of 64, for 50 epochs on a single A100 GPU. Early stopping is applied, with a learning rate of 5e-4 and batch size of 8. AdamW is configured with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and weight decay of 1e-6.

Medical Image Fine-tuned Segmentation.
For segmentation tasks on the RSNA (Shih et al., ), SIIM (Steven G. Langer & George Shih, ), and Covid-19 Rural (Wang et al., ) datasets, we fine-tune both the pre-trained vision encoder and decoder. Training is performed with early stopping at 50 epochs, using a learning rate of 2e-4 and weight decay of 0.05. AdamW is the optimizer, with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. Batch sizes are 8 for SIIM and 16 for RSNA. All configurations follow the protocol from (Huang et al., ).

## Appendix B Extra Visualization

Distribution of Synthetic and Real Data.
We illustrate the distribution of synthetic and real data in Fig 5. For visualization, we use RAD-DINO (Pérez-García et al., ) to extract image features and Med-CPT (Jin et al., ) to extract report features. We then apply Principal component analysis (PCA) to reduce the feature dimensions and visualize the first principal component. As shown in Fig 5, the synthetic data covers a broader range than the real data, indicating greater diversity. In contrast, the real data shows a more concentrated distribution, which may limit the generalizability of MedVLP models.

Pipeline of Synthetic Report Generation. The pipeline for generating synthetic reports using LLMs and balanced sampled clinical entities is illustrated in Fig 6.

Entities Distribution. We visualize the distribution of each type of entity in the MIMIC-CXR dataset. Due to space constraints, only the top 200 most frequent entities are shown, revealing a clear long-tailed distribution in Fig 7, 11, 9, 8, and 10.

Figure: Figure 5: Distribution of Synthetic and Real Data. (a): Comparison of the first principal component distribution of features extracted from RAD-DINO for synthetic and real images. (b): Comparison of the first principal component distribution of features extracted from Med-CPT for synthetic and real reports.
Refer to caption: extracted/5934507/dist.png

Figure: Figure 6: Pipeline for generating synthetic reports. The process begins by generating the ‘FINDINGS’ section, followed by summarizing it into the ‘IMPRESSION’ section. Both sections are checked to ensure they contain the specified entities; if not, the generation process is repeated. The final dataset includes 200,000 synthetic reports, each containing both ‘FINDINGS’ and ‘IMPRESSION’ sections.
Refer to caption: extracted/5934507/reportgen.png

Figure: Figure 7: Top 200 most frequent abnormality entities.
Refer to caption: extracted/5934507/ABNORMALITY.png

Figure: Figure 8: Top 200 most frequent non-abnormality entities.
Refer to caption: extracted/5934507/NON-ABNORMALITY.png

Figure: Figure 9: Top 200 most frequent disease entities.
Refer to caption: extracted/5934507/DISEASE.png

Figure: Figure 10: Top 200 most frequent non-disease entities.
Refer to caption: extracted/5934507/NON-DISEASE.png

Figure: Figure 11: Top 200 most frequent anatomy entities.
Refer to caption: extracted/5934507/ANATOMY.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: While the improvement shown in the paper is quite impressive and the reviewer actually agreed with the authors about the major conclusion, there are still some major concerns about the proposed data synthetic pipeline, as mentioned above. Also, the reviewer has the following questions:

1. The reviewer wonders if it is possible to conduct any form of data quality evaluation on the synthetic data. Even if the experiment results prove that using the synthetic data has greatly improved the performance, it remains unknown if the incorrect synthetic data has injected incorrect knowledge into the model, considering the loose data generation process.
2. Can the author provide some examples of the generated data in the appendix or supplement? The current paper has no example of the generated data, which makes it less convincing.
3. The reviewer also wonders how well the models may perform if they were trained with the **cleaned** MIMIC-CXR as mentioned in the paper. It seems that this cleaned version of MIMIC-CXR is only used for filtering out low-quality synthetic images.

The reviewer sincerely looks forward to hearing from the author during the rebuttal and would be happy to discuss the concerns mentioned above.
soundness: 2
strengths: 1. The proposed image-report generation pipeline uses entities extracted from the real-world report. By sampling these entities with a balanced prior, the synthetic dataset ensures a more balanced distribution compared with real-world data. This has proven to be helpful in the downstream evaluations.
2. This work has conducted a very thorough evaluation with carefully designed experiment settings. It has evaluated the pre-trained model with different training data on multiple downstream, OOD datasets under different settings. It has also evaluated the influence of using different report generation models and text-guided image generation models, which further validate the design of the proposed pipeline.
3. The paper is well-written and easy to follow, and all the technical details are very clear.
summary: This paper provides a thorough evaluation of the medical visual language model with synthetic data. It is proven that training MedVLM with synthetic data and a carefully designed data generation pipeline can significantly boost the model's performance in multiple downstream tasks. The proposed chest X-ray generation pipeline can generate high-fidelity image-report pairs with the given entity. With proper design, the synthetic data can avoid the data quality issues within the real-world data such as long-tailed distribution, and incorrect or low-quality images.
weaknesses: 1. One of the major concerns of this paper is the entity sampling process during report generation. According to the paper, the only constraint on this process is the balanced distribution constraint. However, it is possible that the method samples contradictive entities. For example, “decreased in size” and “increased in size” are two of the top 50 entities in the abnormality entities according to Figure 6. They naturally contradict, but it is possible that the model samples both entities during report generation. Moreover, The quality of the extracted entity is also not very satisfying. There are multiple repeated, or less meaningful entities in the top 200 examples according to the appendix. What makes it worse is that the only quality control on report generation is the evaluation of whether the sampled entity exists or not, rather than if they are reasonable as set. Also, enforcing all the sampled entities to be present in the “Impression” section does not make much sense as well. The “Impression” section in the real-world report may not always contain all the findings. So the quality of the generated report is very concerning without further evaluation, especially considering that there are no example reports provided here. Though it indeed improved the performance.
2. Another secondary concern is about the image quality generated from the synthetic reports. Though the author used a SoTA CXR generative model validated by “clinical experts”, the concerns about generated image correctness still remain, especially considering there might be self-contradicting/incorrect synthetic reports. As a universal concern about the medical image generation method, the method adopted here has no control over the anatomy structure, which means it could generate unrealistic CXR and therefore pollute the downstream training process. 
3. Additionally, there are some implementation details missing in the paper. a) The actual value of $k$, $m$, and the sampling threshold $\tau_{max}$ were not mentioned in the paper or appendix, which is very critical to understanding the method. b) There is no specific description of the pre-training setting with mixed data, where the dataset size is doubled, It is unknown if the training time is also doubled or not.
4. Considering that the paper itself focuses on evaluating the influence of synthetic data on MedVLM, 2 baseline methods are not very convincing. Except for the methods mentioned in the paper that are not suitable, the reviewer would still recommend evaluating more baselines like naive CLIP and MGCA[a]. It is necessary to conduct further evaluation to validate the conclusion of this paper.
5. There are some small typos in the paper such as a) Some of the results in Table 3 row 3 (ConVIRT-MIX) are bolded incorrectly. b) The “Medical Image Fine-tuned Classification” section in section A.2 was aligned incorrectly as well. c) Also, it would be better if the author could annotate the exact numbers in Figure 3. The current version is not very clear, especially in the bottom right figure.


### Reference 
 - [a] Wang, Fuying, et al. "Multi-granularity cross-modal alignment for generalized medical visual representation learning." Advances in Neural Information Processing Systems 35 (2022): 33536-33549.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 1
questions: See weakness part and please answer the listed question and concerns
soundness: 3
strengths: •	Fully synthetic image generation might open the door to more balanced datasets for training larger vision models, which is a challenging topic in medical image analysis.
•	The evaluation presented is extensive and covers many different aspects of image analysis in the medical domain
•	The work delves into recent trends in AI, where pre-trained services are connected to create outcomes beyond their original intended usage (i.e., using pretrained LLMs, NER algorithms, and TTI synthesis…).
•	The emphasis on only using open-source models is commendable.
summary: This paper details a pipeline for generating fully synthetic datasets with the downstream objective of pretraining large vision-language models (VLMs). The synthetic datasets are generated semi-automatically by producing a set of entities using RaTE, a named entity recognition algorithm, which are sampled to produce a subset of findings per sample and are then used to generate a textual report. The textual report is iterated with LLMs to ensure congruence with the initial subset of entities, and is then used for the generation of synthetic images using a text-to-image algorithm. A series of datasets are created using this methodology, which are then explored in a battery of different evaluation strategies for zero-shot classification & segmentation via grounding and further fine-tuning with real data to improve initial VLM performance. The extensive numerical results show consistent performance improvement in VLM when training with mixed real and synthetic data.
weaknesses: •	The evaluation as expressed in the document is rather confusing, where in many cases it’s not clear which is the evaluation/test set used to derive a set of results (e.g., Table 4, Fig 3).


•	There is a lack of a discussion section, which affects the completeness of the work. It would be preferred to move certain aspects of the methodology (and even some results!) to the appendix (e.g., the specific prompts used to identify and filter low-quality images) and create a proper discussion. The reason behind this is because there are some aspects of this work which puzzle the reader (e.g., the low F1 scores in most zero-shot evaluation techniques) that could/should be addresed in the discussion.


•	Section 3.2. could be worded better, so that it’s clear that a single sample contains both anatomical and clinical entities. I found it slightly confusing


•	There are many aspects of this work that would need remediation or discussion. In no particular order: 

1.	In the “analysis” section, there is no description of which test set is employed, so these performance reports are difficult to interpret, validate and compare. 

2.	Section 5 should be merged with 4.2. to create a singular “results” section.

3.	Why do you claim “Some methods also leverage external datasets to boost performance, raising concerns about generalizability”? No metric has been provided. 

4.	Why do you claim that “Our results confirm that many images in the dataset exhibit these issues” in section 3.1.? No metric has been provided. 

5.	Much of the evaluation focuses on classification – what are we to gain using a VLM as opposed to a self-supervised CXR model like Rad-DiNO? 

6.	You’ve identified 150k+ entities, but the synth dataset is 200k. This would lead to the generation of a very low number of samples-per-entity. This can also lead to illogical combinations (e.g., pneumothorax in rib) which AFAIK are not accounted for. 

7.	The generation efforts described in 3.2. look quite compute-intensive. It’s not essential for the work but I’d be interested to have some approximate time taken to generate a single sample (i.e., text generation + filtering and QA + IMPRESSION generation + TTI). 

8.	This methodology could easily lead to collapse because of feeding an algorithm with synth data, see 10.1038/s41586-024-07566-y

9.	The F1 and Dice/IoU scores are quite low in most zero-shot evaluation techniques (tables 1 and 2). It could use some discussion/comparison to other vanilla VLMs and non-VLMs (e.g., zero-shot segmentation with Rad-DiNO). Moreover, the IoU increase wrt the real baseline is not super significant.

10.	It’s not clear how exactly the models are fine-tuned in Section 4.2. What’s the pretraining dataset? Which is the dataset used for fine-tuning?

11.	There’s no comparison to other concept generation or filtering techniques, which is the main contribution of your work. You provide a large amount of experimental runs for downstream applications but the bulk of your work is the synthetic data generation pipeline. Although this is understandable because a paper has to be constrained somehow, some mention in discussion/future steps would be required.

12.	Section 5 contains claims that have not been formally and are hipotheses, e.g.: “likely due to the persistent long-tailed distribution, as the synthetic reports are generated based on real images” or “as synthetic images can be curated using our image filtering procedure to ensure high quality, avoiding issues commonly found in real datasets”.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - The authors noted that 1,448 + 5,512 samples were removed from the MIMIC-CXR dataset; however, this constitutes only about 3% of the entire dataset, raising questions about the necessity of this step. Could the authors provide evidence demonstrating that excluding these ~7,000 samples significantly benefits MedVLP training?
- Incorporating more modalities and tasks would strengthen the argument that synthetic datasets can offer substantial benefits across a broader range of applications.
soundness: 2
strengths: - The authors developed SYNCXR, a large-scale synthetic X-ray dataset, demonstrating that models trained on this dataset could achieve improved performance. 
- They also identified common issues prevalent in real-world datasets, such as MIMIC-CXR.
summary: Given that recent large visual language models require extensive medical data and often rely on synthetic data, the authors question whether synthetic data alone could meet these needs effectively. Through experiments conducted on a synthetic chest X-ray dataset, they found that the synthetic data outperformed real data.
weaknesses: - Concerns with the Experiment and Scope: The focus of this study should be limited to evaluating whether a synthetic CXR dataset can enhance VLM training. Currently, the authors have not included experiments involving other modalities, which limits the broader applicability of the findings.
- RadGraph and RaTE Discussion: I recommend addressing the limitations of RadGraph and the rationale behind RaTE in the related work section, rather than in the main text, as it detracts from the primary focus.
- Novelty and Methodological Concerns: This work primarily takes an engineering approach to address a specific task, lacking novel theoretical contributions or robust validation. For instance, the authors did not provide a mathematical explanation as to whether synthetic data can lead to improved representation learning, nor did they validate the generated data with real clinicians. While model-based validation is acceptable, practical relevance in radiology requires clinician verification.
- For medical image segmentation, the authors should include qualitative comparisons across methods to strengthen their claims, especially as the current quantitative results are presented as single-digit figures, which may be insufficient to persuade readers.
- Additional experiments are recommended to expand the study’s scope, including evaluations across other modalities, tasks beyond visual language pretraining, and models that confidently showcase quantitative results.
- There is a typo in line 233.