Enhancing Plaque Segmentation in CCTA with Prompt- based Diffusion Data Augmentation
Authors: Ruan Yizhe
Xuangeng Chu
Ziteng Cui
Yusuke Kurose
JUNICHI IHO
Yoji Tokunaga
Makoto Horie
YUSAKU HAYASHI
Keisuke Nishizawa
Yasushi Koyama
Tatsuya Harada
Venue: Accepted by TMLR ()

Abstract: Coronary computed tomography angiography (CCTA) is essential for non-invasive assessment of coronary artery disease (CAD). However, accurate segmentation of atherosclerotic plaques remains challenging due to data scarcity, severe class imbalance, and significant variability between calcified and non-calcified plaques. Inspired by DiffTumor’s tumor synthesis and PromptIR’s adaptive restoration framework, we introduce PromptLesion, a prompt-conditioned diffusion model for multi-class lesion synthesis. Unlike single-class methods, our approach integrates lesion-specific prompts within the diffusion generation process, enhancing diversity and anatomical realism in synthetic data. We validate PromptLesion on a private CCTA dataset and multi-organ tumor segmentation tasks (kidney, liver, pancreas) using public datasets, achieving superior performance compared to baseline methods. Models trained with our prompt-guided synthetic augmentation significantly improve Dice Similarity Coefficient (DSC) scores for both plaque and tumor segmentation. Extensive evaluations and ablation studies confirm the effectiveness of prompt conditioning.

Published in Transactions on Machine Learning Research (09/2025)
Enhancing Plaque Segmentation in CCTA with Prompt- based Diffusion Data Augmentation
Yizhe Ruan' Xuangeng Chu', Ziteng Cui', Yusuke Kurose! Junichi Iho", Yoji Tokunaga², Makoto Horie,, Yusaku Hayashi Keisuke Nishizawa?, Yasushi Koyama* Tatsuya Harada
'The University of Tokyo
'RIKEN Center for Advanced Intelligence Project
"Sakurabashi Watanabe Advanced Hcalthcare Hospital
ruanyizhe@mi.c u-u.ooko..ar ac.
Reviewed on OpenRevicw:
Abstract
Coronary computed tomography angiography (CCTA) is essential for non-invasive assess- ment of coronary artery discase (CAD). However, accurate segmentation of athcrosclerotic plaques remains challenging due to data scarcity, severe class imbalance, and significant vari- ability between calcified and non-caleified plaques. Inspired by DiffTumor's tumor synthesis and PromptlR's adaptive restoration framework, we introduce PromptLesion conditioned diffusion model for multi-class lesion synthesis. Unlike single-class methods, our approach integrates lesion-specific prompts within the diffiusion generation process, en- hancing diversity and anatomical realism in synthetic data. We validate PromptLesion on a private CCTA dataset and multi-organ tumor segmentation tasks (kidney, liver, pan- creas) using public datasets, achieving superior performance compared to baseline methods. Models trained with prompt-guided synthetic augmentation significantly improve Dice Similarity Coefficient (DSC) scores for both plaque and tumor segmentation. Extensive evaluations and ablation studies confirm the effectiveness of prompt conditioning.
prompt-
Introduction
Coronary artery disease (CAD) is leading cause of mortality worldwide, significantly affecting global death rates Mensah et al. (2023). Coronary computed tomography angiography (CCTA) is widely used as a non- invasive imaging modality for detecting and characterizing coronary plaques, offering a strong alternative to invasive angiography Marano al. (2020): Serruys et al. (2021). ). Accurate segmentation of plaque regions in CCTA is crucial for quantilying disease burden and guiding interventions.
However, this task is hampered by high inter-class variability between plaque types (calcified vs. calcified plaques) and limited annotated datascts Huang et al. (2020): Lin et al. (2022); Ruan et al. (2025). Moreover, class imbalance is severe since healthy vessel pixels vastly outnumber plaque pixels, which can cause segmentation models to biased toward normal anatomy.
Recent advances in deep lcarning baasin segmentation have achieved promising results in medical imag- ing. Convolutional architectures like U-Net. Ronneberger al. (2015) and its self-configuring variant. nnU- Net Isensee et al. (2018) have become standard, and transformer-based models (e.g. TransUNet Chen et al. (2021h) further improved multi-organ segmentation by capturing long-range context. Nevertheless, these models require large-scale annotated data to generalize well, making data augmentation essential to alleviate data scarcity. Traditional augmentation techniques (random flips, rotations, etc.) provide limited benefit. as
https://github. com/RuanYizhe--ee cttp//uaariiiaaeee/////pppaaaaaa

---

Published in Transactions on Machine Learning Research (09/2025)
they do not introduce new anatomical patterns or pathology variations. This limitation motivates the use of generative models to create realistic synthetic lesions and images for augmentation. Generative adversarial networks (GANs) Goodfellow et al. )) Armanious et al. (2020); Wu et al. (2024) and variational autoen- coders Pesteic et al. (2019) Esser et al. (2021) have been explored for medical data augmentation, but can sulfer from mode collapse and limited diversity. Recently, diffusson models have emerged as a powerful class of generative models capable of producing high-fidelity images by iterative denoising. In medical imaging, diffusion models have shown promise in synthesizing anatomically realistic structures. For example, [Chen et al. 2024) introduced DiffTumor, diffusion- based framework for realistic tumor synthesis across organs, demonstrating improved generalization over GANs. However, DiffTumor is restricted to single-class lesion generation, focusing on one tumor type at a time. This limits its applicability for tasks where multiple lesion classes coexist.
Inspired by PromptIR's success Potlapalli et al. (2023) adaptive image restoration using task-specific prompts, we propose integrate prompt-based conditioning into a diffusion model for multi-class lesion synthesis. Our method, termed PromptLesion for multi-class lesion generation, incorporates learnable lesion- specific prompts during the diffusion generation process. By conditioning target lesion type, our model can synthesize both calcified and non-calcified plaques in Coronary Computed Tomography Angiography (CCTA) within a single framework. This controlled augmentation addresses the data imbalance by generating diverse examples of under-represented classes. We extend this approach beyond CCTA plaques to multi-organ tumor synthesis: by conditioning on a lesion-specific prompt (e.g. kidney vs. liver tumor, carly vs. late stage), we enable a unified model to generate a variety of tumor types. Our hypothesis is that prompt-guided diffusion augmentation will produce more diverse and realistic lesions, leading to improved segmentation performance on both plaque and tumor segmentation tasks. We validate this by training segmentation models on: (a) real data only, and (b) real + syuthelic augmented data, comparing outcomes. Experiments on a private CCTA dataset show that adding synthetic plaques yields notable DSC improvements for plaque segmentation. Similarly, on public kidney Heller et al. (2020). liver Bilic et al. (2023), and pancreas Antonelli et al. (2022) CT datasets, our multi-class synthetic tumors improve segmentation accuracy across all organs, outperforming the DiflTumor augmentation approach.
prompt that encodes the
In summary, our work makes the following contributions:
Prompt-conditioned diffusion augmentation: We introduce PromptLesion, the first diffusion- based framework integrating lesion-specific prompts for controlled multi-class lesion synthesis. This design provides an easy-to-scale solution adaptable to multiple lesion classes and anatomies.
Enhanced segmentation via synthetic data: Our syuthetic augmentation method significantly boosts segmentation performance, effectively addressing class imbalance by oversampling rare lesions realistically.
Generalization across anatomies: We validate PromptTesion across CCTA plaque and multi- organ tumor datasets (kidney, liver, pancreas), achieving superior segmentation accuracy and lesion diversity compared to baseline methods.
This paper is structured as follows: Section 2 reviews related work on medical image synthesis, diffusion models, and prompt-based learning. Section 3 details our prompt-conditioned diffusion methodology, in- cluding the VQGAN-based autoencoder and multi-class prompt integration. Section 4 presents experiments CCTA and lumor datasels, with comparisons to baselines and ablations. Section 5 concludes with key findings and future directions.
2 Related Work
2.1 Medical Image Synthesis and Augmentation
Data Augmentation in Segmentation: Data scarcity and class imbalance are common challenges in medical image segmentation, and various augmentation strategies have been proposed. Beyond basic transformations,
2

---

Published in Transactions on Machine Learning Research (09/2025)
rescarchers have explored generating synthetic data to enrich training sets. Generative Adversarial Networks (GANs) have been widely used for medical image synthesis and augmentation. For example, Arir et al. 2018) used GANs to generate synthetic liver lesions in CT scans to improve tumor classification performance. Bowles et al. (018)) applied GAN-based augmentation for brain lcsion segmentation, reporting improved segmentation accuracy. In digital pathology, Xue el al. (2021) introduced HistoGAN, a conaltional GAN that synthesizes histopathology image patches conditioned on cancer class, yiclding significant gains in classification of rare cancer subtypes. These works demonstrate that synthetic lesion data can effectively mitigate class imbalance and improve model generalization in various medical imaging contexts. However, most GAN-based approaches are class-specific they train separate models for each lesion type or organ. This limits scalability and fails to exploit potential commonalities between classes. Moreover, GANs can struggle to maintain anatomical realism for complex structures. These limitations motivate the use of diffusion models, which offer improved stability and diversity.
2.2 Diffusion Models for Medical Image Synthesis
Diffusion probabilistic models have recently shown impressive results in image generation. A diffusion model gradually adds noise to training images and learns to reverse this noising process, enabling it to sample realistic images from pure noise. Ho al. (2020) introduced the DDPM formulation and demonstratod high-quality image synthesis on natural images. Due to their capacity for generating fine details and mode coverage, diffusion models are well-suited to medical image synthesis, where preserving anatomical fidelity is critical. Several works have adapted diffusion models to medical tasks. Diffusion-bass based augmentation was applied by Chen et al. (2024) ) in DiffTumor, which synthesized tumors on CT and improved cross- organ tumor detection. Their diffusion-generateed tumors enabled the training of a segmentation model that generalized to multiple organs, highlighting diffusion models' potential in biomedical data augmentation. Kazerouni et al. (2023) survey diffusion models in medical imaging and note their success in anomaly de- tection and modality translation tasks. Zhang et al. (2024) propose DiffBoost, a controllable text-guided diffusion model for medical image segmentation augmentation. By incorporating structural edge information and textual labels, DiffBBoost generates diverse synthetic images that improve segmentation performance on ultrasound, CT, and MRI datasets. [Dojsembe et al. (2024) introduce Med-DDPM, a 3-D diffusion model that synthesises anatomically faithful brain-MRI volumes from tumour masks, further un- derscoring conditional diffusion's value for that diffusion-baser synthetic data can significantly boost downstream segmentation accuracy. Compared to GANs, diffusion models offer improved mode diversity and training stability at the cost of higher compu- tation. Our work builds upon these advances by introducing conditional diffusion for multiple lesion classes simultaneously. While diffusion models have been successfully applied to lasks such as anomaly detection and single-class lesion synthesis (e.g., for a specific tumor type), many recent works also explore domain translation, denoising, reconstruction, and multi-class tasks. Nonetheless, the majority of lesion-focused diffusion studies to date often concentrate on one particular lesion type rather than simultancously address- ing multiple classes; we extend the paradigm to multi-class lesion generation via prompt embeddings, as described next.
data augmentation. These results reinforce
2.3 Prompt-based Learning in Generative Models
Prompt learning- - providing additional conditioning inputs or context to guide model behavior has gained traction in both NLP and vision Brown al.) (2020); Liu et al. (2023); Shin al. (2020); Radford et al. (002 0: Zhou et al. (2022): Jia et al. (2022). In NLP, prompt templates are used to steer large language models to perform specific tasks without fine-tuning. In computer vision, learnable prompts or tokens have been employed to adapt transformers to new tasks (e.g. LoCoOP Miyai et al. (2023) for few-shot classification). Prompt-based techniques have only recently been explored in image generation and restoration. Pollapalli et al. 22023 introduced PromptIR, which uses trainable prompt vectors to encode degradation information (noise, blur, haze levels) for image restoration. By injecting prompts into a restoration network, Prompt achieved state-of-the-ar results on multiple degradation types with a single model. This demonstrated that wight prompts can effectively modulate deep network's behavior based on task-specific context. Tn
3

---

Published in Transactions on Machine Learning Research (09/2025)
the generative domain, approaches like text-to-image diffusion Zhang et al. (0223) allow controlling outputs via textual prompts. Yet, prompt-based conditioning in medical generative models remains underexplored.
PromptIR's plug-aud-play prompt modules inspire our work: we design a prompt-based conditioning mech- anism for a diffusion generator. The prompts in our model encode lesion type information (plaque or tumor category) and guide the image synthesis process to produce the desired abnormality. This is conceptually related to conditional generation but with flexible learned prompt vector instead of a simple one-hot label or fixed scalar conditioning. The prompt is injected at multiple stages of the diffusion U-Net, allowing it to inftuence the denoising trajectory towards the target lesion appearance. We show that such prompt condi- tioning yields distinct benefits in multi-class setting, it provides a form of adaptive guidance that can be tuned for each lesion class without retraining the entire model.
2.4 Applications of Multi-Class Lesion Synthesis
While single-class synthetic augmentation has demonstrated benefits in medical imaging contexts, such as liver lesions Frid-Adar et al. (2018), brain lesions Bowles et al. (2018), and pathology classification Xue
et al. (2021). the capability to simultaneously handle multiple lesion classes within a single framework is
becoming increasingly crucial. Tn clinical practice, multiple lesion types or pathologies frequently coexist
within single imaging scan, such as varying plaque compositions in coronary arteries Ruan al. (0025)
melastalic lumors distributed across multiple organs. Nevertheless, prior generalive augmentation mods including GAN-based techniques Armanious et al. (2020); Wu et al. (2024) and diffusion baaudd methods like DiffTumor Chen et al. (2024), typically focus on synthesizing a single lcsion class per model. This single-class strategy does not effectively exploit shared anatomical or pathological characteristics, leading to inefficiencies and limited scalability when expanding to multiple lesion types.
Our work addresses this limitation by introducing a unified, prompt-driven diffusion framework designed explicitly for multi-class lesion synthesis. We target two distinct clinical domains: coronary artery plaques (calcified and non-caleified) and abdominal organ tumors (kidney, liver, pancreas, each with early and late stages). In the context of coronary computed tomography angiography (CCTA), our method synthesizes both plaque types within the arterial context, providing balanced lcsion diversity that enhances segmen- tation robustness on scans with mixed plaque compositions. Similarly, for tumor segmentation tasks, our approach generates diverse tumors across different abdominal organs, allowing segmentation models to learn generalized lesion features rather than becoming overly specialized on a single organ or lesion class.
Leveraging prompt-conditioned diffusion ensures anatomical coherence in synthesized images despite varia- tions in lesion appearance. Consequently, we produce richly augmented datascts that encompass multiple lesion categories, significantly improving segmentation accuracy across lesion classes. Our experimental re- sults consistently demonstrate superior Dice similarity coefficients for both plaque and tumor segmentation tasks compared to baseline methods trained solely on real data or single-class augmentation strategies. Over- all, prompt- based multi-class lesion synthesis provides a scalable, generalizable, and effective augmentation strategy, representing a significant advancement in generative augmentation for medical imaging.
3 Method
Our proposed method combines latent diffusion model with prompt-based multi-class conditioning to generate realistic lesions for data augmentation. The overall pipeline (illustrated in Figure I consists of three stages: (1) a VQGAN autoencoder that learns a compact latent representation of medical images, (2) a conditional diffusson model operating in the latent space, guided by lesion prompts to produce synthetic abnormalities, and (3) a segmentation model to evaluate the performance of synthetic images. We describe each component and its integration below.
3.1 VQGAN-based Reconstruction
Following the approach introduced by DimoTumor Chen et al. (2024), we employ a Vector Quantized Gen- erative Adversarial Network (VQGAN) Faser et al. (2021) to encode medical images into a compact latent
4

---

Published in Transactions on Machine Learning Research (09/2025)
Step1. Autoencodar Model
VODAN
Stop2. PromptLosion
Bood
Lasion Lobel
redls
Step3. Segmentation Model
Prieeee
Ledter libe
Clobal Anerage
Operation
Unee
Chanapi-wle
Settatterkknn
Figure Overview of our Prompt-based Diffusion augmentation framework. (a) Step VQGAN Autoen- coder encodes input medical images into compact latent representations, (b) Step 2: Prompt-conditooed diffusion model named PromptLesion synthesizes lesion-specific latent codes guided by prompt embeddings. (c) Step 3: Segmentation Model evaluates the performance of synthetic lesion images. See Section 3 for detailed descriptions.
space, as step 1 in Figure L shows. Working in this latent representation significantly reduces computa- tional demands and enables the diffusion model to focus better on meaningful anatomical and pathological structures.
Specifically, given an input 3D medical image patch x E R the encoder Es produces a compressod latent representation
(x) Rhxxxd Fach element of is subsequently quantized by mapping it to the nearest vector from learned discrete codebook Z = [e,)K,,, yielding the quantized latent code The decoder Go then reconstructs the image from this quantized code, resulting in the reconstructed output I = Gz(2).
To train VQGAN effectively, we optimize a combined loss objective comprising several terms: a pixcl-level Lı reconstruction loss (Cree) ensuring accurate overall reconstruction, a perceptual loss (Cperceptuat) to preserve anatomical details across axial, coronal, and sagittal planes, and two codebook-reeated losses codebook optimization loss (Loodebook)) and encoder commitment loss (Ccommit)-wee which collectively stabilize quan- tization training. Additionally,
introduce adversarial training with two discriminators: volumetric discriminator (D,) enforcing overall 3D anatomical coherence and a slice-based discriminator (D.) improv- ing slice-wise realism. We further incorporate discriminator based feature matching loss (Ematoh) to guide the decoder toward genorating realistic intermediate ropresentations. The discriminator loss (Ldisc) is thus formulated as:
Lasac log (2)o(x) + log (1 Da)o
(1)
After training, encoder and decoder parameters are frozen. The diffusion model operates exclusively on this discrete latent space, ensuring efficient synthesis while retaining critical anatomical information.
3.2 Diffusion Model with Prompt Conditioning
Inspired by the approach outlined DiffTumor Chen et al. (2024). we employ Denoising Diffusion Proba- bilistic Model (DDPM) to synthesize latent representations containing realistic lesions, as step 2 in Figure shows. Specifically, define diffusion model using U-Net architecture fo I, m, hhealthy .p) designed
5

---

Published in Transactions on Machine Learning Research (09/2025)
to reverse predefined forward noising process. Note that, similarly to Dimor, the diffusion model is conditioned on both the lesion mask m and the healthy region ghealthy = (1 m) Ô z. During the forward process, Gaussian noise is progressively introduced into the initial latent representation 20 (obtained from real images via the trained VQGAN encoder), resulting in a noisy latent code 24 at timestep E [1,7]. The U-Net then predicts the added noise given Et, timestep i, lesion mask m, healthy CT region shealthy and a lesion-specffic prompt embedding p.
We explicitly introduce prompt conditioning to guide lesion synthesis, integrating prompts at multiple scales within the U-Net architecture. Prompts encode lesion class labels(one-hot) into dense vectors us- ing a prompt embedding network (e.g. [0, 1, for calcified plaque, [0,0, for non-calcified plaque in CCTA datasets;[0, 1, 0, 0, O, 0, 0J for early stage tumor in liver datasets). Our framework further supports controlled multi-class lesion synthesis via user-specified prompts (Section3.2.1) and leverages adaptive prompt learn- ing through implicit prompts derived from a transformer-bass encoder to enrich contextual relationships among lesion types (Soction3.2.2). Formally, the diffusion model is trained by optimizing the standard DDPM objective, minimizing:
Ezn, / [|e [0(=,, L, m, 2 heatthy, .p)]]]
(2)
where [0[=, I, m, phealtyy, p) is our PromptLesion Network, as a 3D U-Net variant with prompt block, in- terleaved self-attention layers and convolutional layers Ho et al. 22000;; ; Nichol & Dhariwal (2021) the noisy latent 2, is obtained through the forward noising step defined by:
Z4 Vānzo + v1 cre, ~0,
(3)
Through this training, the U-Net learns to progressively denoise latents towards plausible, prompt-specifi anatomical structures and lesions.
3.2.1 Multi-class Lesion Generation via Prompts
Once trained, our diffusion model enables controlled synthesis of lesions conditioned on uscr-speeified prompts. For instance, to generate synthetic coronary plaques, we first sample random Gaussian la- tent 2T ~ (0, I) and select desired lesion class Cplaque E calcified, non-calcified The prompt embedding network generates the corresponding prompt vector p. We then perform iterative reverse denoising steps, computing:
z = fo(z,, 1, m, healthy .p), T=,,T-1,.....
ultimately yielding a clean latent 20 reprosentative of vessel containing the dosired plaque type. Decoding #o via the pretrained VQGAN decoder produces synthelic but anatomically coherent CCTA images suitable for segmentation training.
A similar procedure is applicd for tumor data, where prompts condition the model to generate organ-specific and stage-specific lumor lesions. Crucially, our unified diffusson model supports generation of multiple lesion types using a single architecture, differentiating lesion classes solely through prompt vectors. The model can also generate lesion-free images by employing dodicated "healthy" identity prompt, enabling versatile data augmentation strategies, including guided insertion of synthetic lesions into real lesion-free backgrounds through partial reverse diffusion steps analogous to latent-space inpainting.
Following DiffTumor Chen et al. (202)), we also adopt lesion-background blending, which they showed is superior to fully synthetic volumes. Our augmentation procedure integrates synthetic lesion patches into real images at anatomically plausible locations guided by clinical domain knowledge, such as inserting synthetic plaques along coronary artery walls.
3.2.2 Adaptive Prompt Learning
To further enhance the informativeness and representational capability of our prompt-based augmentation framework, we extend the prompt encoding mechanism beyond simple onc-hot vectors by incorporating im- plicit prompts extracted through a lightweight transformer-base encoder, as shown in Figure Specifically, instead of directly using one-hot encoded vector of lesion labels explicit conditioning input, we employ
6

---

Published in Transactions on Machine Learning Research (09/2025)
W
Block
W
muiti- channel mask
W4:
Transformer
W/x
Prompt Components
Linear Combination
Channel-wise Combination
Pooling
Operation
C
Convolution
Convolution
Figure 2: Transformer-baset Prompt Network: Instead of conditioning with one-hot vector, transformer- based prompt network conditions with multi-channel mask.
small transformer encoder Vaswani et al. 2017 to embed the multi-class segmentation masks into latent embeddings. These embeddings inherently capture richer contextual relationships among different lesion classes, thus serving as more informative implicit prompts to guide the diffusion process. The lightweight transformer encoder consists of 3 layers, each comprising multi-head selfattention and position-wise feed- forward networks, effectively balancing representational capacity and computational efficiency. Compared the explicit one-hot encoding used previously, this implicit prompting strategy leverages spatial and class relational context, potentially improving the generation fidelity and diversity of synthetic lesions, demonst.rate empirically in subsequent experiments.
3.3
Segmentation Model
As shown in step 3 of Figure 1 to evaluate our synthetic augmentation, we use standard segmentation models including 3D U-Net and nnU-Net, following the general methodology of DiffTumor Chen et al. (2024). Specifically, for the CCTA plaque segmentation task, we combine synthetic plaque-containing slices generated by PromptLesion with healthy artery slices from the real training dataset. Each synthetic slice includes automatically generated lesion masks indicating calcified or non-calcified plaques, thus enriching lesion classes while maintaining realistic artery anatomy.
For the multi-organ tumor segmentation tasks (kidney, liver, pancreas), directly follow DiffTumor's procedure, integrating real tumor data. These segmentation models serve as evaluation tools to measure the effectiveness of our synthetic data- by comparing models trained with and without PromptLesion augmentation, we assess whether the generated images contribute to improved performance on rewmstream segmentation benchmarks.
wew.-Lsiionnnnnrrtteee synthetic tumor images and their corresponding masks with
4 Experiments
4.1
Datasets
Our method was trained and evaluated on two primary tasks: plaque segmentation in CCTA and tumor segmentation in abdominal CT.
Private CCTA Dataset. For plaque-related experiments, we used a private of 100 cardiac CTA volumes acquired at the Sakurabashi Watanabe Advanced Healthcare Hospital. Following the data split
7

---

Published in Transactions on Machine Learning Research (09/2025)
ratio of prior work Ruan et al. 2025), the cohort was divided into training set of 70 patients, validation set of 10, and a test set of 20. The annotation procedure was carried out by cardiac experts. For each CCTA volume, the experts provided detailed, bliccc segmentation masks using specialized cardiac amalysis software. These manual annotations delincated five distinct classes: background, artery lumen, artery wall, calcified plaque, and non-calcified plaque. To ensure the quality and consistency of the ground truth, the annotation process involved multi-stage review where each segmentation was cross-verified by second senior expert. This study was conducted with IRB approval granted by the Ethies Committee of RIKEN (No. Wako2022-14), and all data was fully anonymized.
Public Tumor Datasets, For tumor segmentation, we used the same public datasets as DiffTumor Chen
et al. (0224): the Kidney Tumor dataset (KiTS) Heller et al. (2020), the Liver Tumor dataset (LiTS) Bilic
et al. 2023), and the Pancreas Tumor datasct (MSD Pancreas) Antonelli et al. (2022) We combined their
training sets to form multi-organ lumor training set and evaluated performance each organ's a slandard
test split.
Pre-training Dataset for VQGAN. Following the procedure in DiffTumor Chen et al. (2024) , the VQGAN autoencoder for the tumor task was pre-trained on the AbdomenAtlas--i 8K dataset Qu et al. (0023 For the plaque task, the VQGAN was pre-trained on our private CCTA training set.
4.2 Implementation Details
Data Curation and Preprocessing. For CT preprocessing, we followed the procedure used in DiffTu- mor Chen et al. (2024), extracting fixed-size 3D sub-volumes from the original CT volumes for training the diffusion model. For CCTA preprocessing, the raw CCTA data was provided as multi-slice TIFF files. We first developed pipeline to convert this data into the Nifti format. For each patient's artery, TIFF slices were stacked to form 3D volume. The original Hounsfield Unit (HU) values were scaled to grayscale inten- sity values in the range [0, 255]. The corresponding label files were converted into multi-class segmentation masks, where specific grayscale values were mapped to integer class labels (e.g., background: 0, lumen: 1, artery wall: 2, calcified plaque: 3, non-calcified plaque: 4).
This curated Nifti dataset then served as the input to our MONAI-based training pipeline. The key prepro- cessing and augmentation steps include: (1) reorienting volumes to RAS coordinate system; (2) resampling volumes to a uniform voxel spacing of [e.g., 1.0 1.0 1.0 mm3]; (3) normalizing intensity values from window of [a min, a max] to [b_ mim, b max]: (4) performing balance random cropping to fixed patch size of 96 96 96 voxels; and (5) applying random 90-degree rotations.
Training. The random seed for all experiments was set to 1234 to ensure reprodueibility. Our training pipeline is implemented using the MONAI framework. To address class imbalance, employed MONAI's class-aware cropping methods, specifically RandCropyyYoslleggabeell and RandCropyyLaeelllaseeed to ensure that training patches were sampled with balanced representation of different lesion types. For data augmentation, we applied only random 90-degree rotations (RandRotate9Od with a probability of 0.1). No other complex augmentations such as affine transformations or elastic deformations were used. Further details on hyperparameters and training schedules are provided in the appendix.
4.3 Benchmarking Protocol for CCTA
Due to patient privacy regulations, CCTA dataset cannot be made publicly available. To facilitate fair comparisons for future research, we propose the following protocol:
File Formats: All inputs and outputs should be in the Nifti (.nii. gz) format.
Evaluation Metrics:
The primary metric the Dice Similarity Coefficient (DSC). Performance
should be reported as the average DSC for each plaque class individually and as mean across all classes.
8

---

Published in Transactions on Machine Learning Research (09/2025)
Standard Preprocessing: External methods should, at minimum, implement standard prepro- cessing steps, including isotropic resampling and intensity normalization, as detailed in our public code.
4.4 Experimental Setup and Baselines
We integrate our synthetic data into the training of segmentation models and compare against multiple baselines:
Baseline segmentation (Real-Only): Train segmentation networks using only the real training images and labels, with no synthetic augmentation, using bascline U-Net and nnU-Net architectures. This represents the standard supervised scenario with limited data.
PromptLesion Augmentation (Ours): Train the same segmentation networks on the combination of real and PrompiL.ssoon-eent
images. For each real image in batch, we add one more synthetic images of under-representer classes (e.g. synthetic non-caleified plaque image if that class is rare) generated by our model. The synthetic images come with automatically generated segmentation masks (since we know the lesion ground-truth in the synthetic world).
PromptLesion" Augmentation (Ours): In the additional experiment, we thoroughly evaluated the impact of using transformer-derived implicit, prompls compared to the conventional one-hot vector approach. We conducted training of the PromptTumor model using the implicit transformer- based prompts under identical conditions as previous experiments. Segmentation results were evaluated on the same tumor datasets (liver, kidney, pancreas) and coronary artery plaque datasets, using baseline U-Net and mnU-Net architectures.
DiffTTumor Augmentation: As an additional baseline, we simulate using DiffTumor for augmentation. DiffTumor can only synthesize syuthetic tumors for each organ separately using DiffTumor's released model and add those to training. This baseline tests if our multi-class approach yields an advantage over using multiple single-class generators.
type of tumor at time (e-g. liver tumors). We generate
Syn Tumor Augmentation: We compare our diffusion- based augmentation with SynTumor Hu et al. (2023). These images augment. the training set similarly for both CCTA and tumor datasels.
DorrDsPMoorrscbbbe et al. (0244) to handle all lesion classes jointly by conditioning on multi-class masks. For each healthy CT, a lesion mask of a target class sampled; Med-DDPM then generates a lesion volume that is composited into the CT following the DiffTumor blending strategy. The resulting synthetic scans and their masks are mixed with the real data using the same sampling ratio as PromptLesion, allowing us assess a mask-conditioned diffusion baseline without prompt embeddings.
Med-DDPM (multi-class) Augmentation: We extend Mcd-DDPM Dorjsembe
Segmentation architectures: We report results with two segmentation models: standard 3D U-Net and the nnU-Net 3D framework. We train each model each training set variant (Real-Only- Training, Mix- Mixinngin, etc). All models are trained for 500 epochs with early stopping the validation set.
Note that we use the DiceCELoss Sudre et al. (0177) for all the segmentation trainings. We emphasize that the synthetic images are used in addition to real images (not. replacing them). The proportion of synthetic data is tuned on the validation set we found using a 1:1 ratio of synthetic: real achieved the best results without overwhelming the network with possibly rodundant synthetic samples.
4.5 Evaluation Metrics and Protocol
Segmentation performance evaluated primarily by the Dice Similarity Coefficient (DSC), which measures the overlap between predicted masks and ground truth We report DSC for each class (calcified vs non- caleified plaque, tumor in each organ) and overall mean DSC. We compare model performance the

---

Published in Transactions on Machine Learning Research (09/202))
CCTA
scan
Ground
truth
Unet
Real
Unet
Ours
Figure 3: Vision Comparison for CCTA datasets: In mask images, black refers to background, red denotes coronary artery wall, green indicates calcified plaque, and purple is non-calcified plaque
independent Lest. sets for each (ask:(a) CCTA Plaque Segmentation: We evaluate on 10 held-out CCTA volumes, computing the per-lesion DSC. (b) Multi-organ Tumor Segmentation: We test on the official test sets of KiTS, LiTS, and MSD Pancreas. Following DiffTumor, we perform evaluation tumor-wise and report the mean DSC across all tumors.
For statistical significance, run each experiment 3 times with different random seeds and report the average DSC.
4.6 Results
Figure FigureBshowss qualitative segmentation results on the CCTA dataset, comparing a baseline model (trained on real data only) with our proposed PromptLesion. Our method effectively captures challenging plaque regions, particularly non-caleified plaques, which often have ambiguous boundaries and relatively low contrast with surrounding tissues. In contrast, the baseline segmentation model often misses these subtle plaque structures or incorrectly segments partial regions, leading to incomplete or noisy segmentations. By leveraging our prompl-conditioned diffusion augmentation, the segmentation predictions become more accurate and closely aligned with expert annotations, demonstrating enhanced sensitivity towards subtle lesions.
Similarly, Figure 4 illustrates visual comparisons on the tumor datascts (kidncy, liver, pancrcas). The model trained with our prompt-based diffusion augmentation produces more accurate tumor masks organs compared to the baseline, especially for smaller less conspicuous lesions. Figure g demonstrates similarly enhanced segmentation accuracy for kidney, liver, and pancreas tumors, highlighting our method's generalizability.
Quantitatively, Table I summarizes DSC results for CCTA plaque segmentation. Our PromptLesion and PromptLesion' methods consistently outperformed the basclines, achieving marked improvements particu- larly for the challenging non-calcified plaques. PromptLesion' provided the best overall performance across both segmentation architectures.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Liver
Kidney
Pancreas
CT scan
Ground
truth
Unet Real
Unet
Ours
Figure 4: Vision Comparison for Tumor Datasets: In mask images, red is the background, purple refers to organ, and the black is the target tumor.
Table 2 compares the segmentation performance of various approaches for multi-class tumor segmenta- tion, evaluated on three different organs (liver, kidney, pancreas) with two segmentation backbones (U-Net and nnU-Net). "Real Tumors" represents training on real tumor data, only (no synthetic augmentation). "DiffTumorasg " are baselines that use single-source training (trained on one organ's data) but are tested across multiple domains (performance averaged across organs). Our proposed methods, "PromptLesion" and "Prompt. consistently outperform these baselines (including DiffTumoravg) in multi-domain segmen- tation without sacrificing performance. The results demonstrate that PromptTumor achieves the best overall performance (see the "Avg" column), highlighting its effectiveness in generaling multi-class lumors across different domains.
4.7 Ablation Study
4.7.1 Multi-class tumor generation across domains
Table3 presents an ablation study evaluating multi-class tumor generation across three domains (liver, kidney, pancreas) using two different backbones (UNet and nnUNet). "DiffTumo"" denotes the baseline method trained on the same source data, while "PromptLesion" (and its variant "PromptLesion represents proposed approach for multi-class generation. The results indicate that PromptLesion achieves comparable or even higher performance than DiffTumor, demonstrating that our prompt-based framework does not sacrifice performance when extending tumor generation to multiple classes.
4.7.2 Prompt conditioning
To systematically investigate the contribution of transformer-based implicit prompts, performed ab- lation study by comparing: (a)slic (Real Data Only): Training segmentation models solely on real (b)iffur)i Synthetic augmentation using the DiffTumor method. (c) PromptLesion (One-hot): Our initial PromptTumor method conditioned on one-hot vectors of target lesions. (d)otptttession'' (Transformer-based Implicit Prompt): The improved method introducing latent emboddings generated from multi-class masks through lightweight transformer encoder. We observed consistent performance boost when transitioning from explicit one-hot prompts to implicit prompts, highlighting that implicit embeddings
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table Quantitative comparison plaque generation in Dice Score.
Backbone
Method
Calcified Plaque
Non-calcified Plaque
Avg
Real CCTA
68.1
21.3
44.7
SynTumor
69.5
27.2
48.4
DiffTumoravy
70.2
30.2
50.2
Unet
Med-DDPM
71.6
34.3
52.9
Prompt.esioo
73.4
38.2
55.8
PromptLesion'
75.1
39.0
57.1
Real CCTA
69.2
23.1
46.2
SynTumor
69.9
28.3
49.1
DiffTumoravg
70.6
31.7
51.1
unUnet
Mcd-DDPM
70.9
35.3
53.1 1
PromptLesion
74.1
40.1
57.1
PromptLcsion'
75.6
44.3
59.9
Table 2: Quantitative comparison of tumor generation in Dice Score.
Backbone
Method
Liver
Kidney
Pancreas
Avg
Real Tumors
35.8
50.0
43.5
43.1
SynTumor
48.3
53.3
44.2
48.6
DiffTumoraeg
52.3
54.2
46.3
50.9
Unet
Med-DDPM
59.2
54.3
52.6
55.4
Lesion
58.6
58.9
52.9
56.8
PromptLesion'
64.0
60.0
53.7
59.2
Real Tumors
47.2
51.3
33.4
43.9
SynTTmmrr
49.2
54.1
36.4
46.5
DiffTumoravg
53.8
55.7
39.8
49.7
nnUnet
Med-DDPM
53.7
55.3
45.3
51.4
PromptLesion
56.0
59.3
45.2
53.5
PromptT.esion
58.2
61.1
46.6
55.3
encode additional structural and semantic information beneficial for the diffusion model. The quantitative evidence in Tables and corroborates this observation, revealing an average DSC gain of approximately 2-3 percentage points both plaque and tumor segmentation benchmarks relative the original PromptLessoon.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Ablation study for multi-class tumor generation across domains
Backbone
Method
liver
kidney
pancreas
Avg
Diffoumm
59.3
60.9
54.2
58.1
Unet
PromptLesion
58.6
58.9
52.9
56.8
PromptLesion
64.0
60.0
53.7
59.2
DiffTumrr
58.9
62.8
44.8
55.5
nnUnet
PromptI.csion
56.0
59.3
45.2
53.5
PromptLesion'
58.2
61.1
46.6
55.3
5 Conclusion
We proposed PromptLesion, a prompt-conditioned latent-diffusion framework that tackles data scarcity and severe class imbalance in coronary computed tomography angiography (CCTA). By injecting lesion-specific prompts into the denoising Irajectory, the model synthesizes anatomically coherent calcified and non-caleified plaques, supplying segmentation networks with hard-to-obtain training samples. Across both backbone architectures, the augmented models consistently outperform the real-data baselines in plaque Dice score, with especially pronounced improvements for the under-represented non-caleified plaques.
To probe generalization, we reused the same framework -haavoin--hnningggg only the prompt definitions on public multi-organ tumor datasets. Consistent Dice improvements on liver, kidney, and pancreas benchmarks con- firm that prompt-guided diffusion scales gracefully unseen anatomies and lesion types without retraining the generator.
Impact: This work advances the state-of-the-art in generative medical image augmentation by introducing an easily scalable, multi-class synthesis paradigm. Climically, enhanced plaque segmentation can improve patient risk assessment in coronary artery disease, and improved tumor segmentation has significant potential for better treatment planning. Our prompt-conditioned diffusion framework can be readily adapted to other medical imaging scenarios by defining suitable prompls, underscoring broad applicability and potential reduce annotation burdens in clinical Al deployments.
Broader Impact and Clinical Considerations
Our PromptLcsion method addresses the challenges of label scarcity and class imbalance by generating realistic synthetic lesions. We outline the following considerations for its
This technology is designed exclusively as training-time tool to enhance model robustness, especially for rare classes. It is not intended for direct diagnostic use, and its synthetic outputs should not replace real clinical data.
Intended Use:
Clinical Risk:
A potential risk involves synthetic lesions containing artifacts that could mislcad models or clinicians. Consequently, rigorous and comprehensive validation is essential before any clinical deployment of models trained with this method.
Data Privacy: The clinical data used in this work fully anonymized under IRB approval. We strongly encourage researchers to apply the same strict privacy standards to their data.
13

---

Published in Transactions on Machine Learning Research (09/2025)
References
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 13(1):1128, 2022.
Karim Armanious, Chenming Jiang, Mare Fischer, Thomas Küstner, Tobias Hepp, Konstantin Nikolaou, Sergios Gatidis, and Bin Yang. Medgan: Medical image translation using gans. Computerized medical imaging and graphics, 79:101684, 2020.
Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor seg- mentation benchmark (lits). Medical image analysis, 84:102680, 2023.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert. Gan augmenta- tion: Augmenting training data using generative adversarial networks. arXiv preprint arXiv:1810. 10863, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan I. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint ar Xiv:22102.. 04306, 2021.
Qi Chen, Xiaoxi Chen, Haorui Song, Zhiwei Xiong, Alan Yuille, Chen Wei, and Zongwei Zhou. Towards generalizable tumor synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11147-11158, 2024.
Zolnamar Dorjsembe, Mising-Kuo Pao, Sodtavilan Odonchimed, and Furen Xiao. Conditional diffusion models for semantic 3d brain mri synthesis. IEEE Journal of Biomedical and Health Informatics, 2877):4084-4403, 2024. doi: 10.1109/ JBIH...2...22333338
Patrick Fsser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the LEEE/CVF conference on computer vision and pattern recognition, pp. 12873 12883, 2021.
Maayan Adir,Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan- based syuthetic medical image augmentation for increased emn performance in liver lesion classification. Neurocomputing, 321:321-333 331, 2018.
Ian J Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Nicholas Heller, Sean McSweeney, Matthew Thomas Peterson, Sarah Peterson, Jack Rickman, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Joel Rosenberg, al. An international challenge to use artificial intelligence define the state-of-the-art in kidney and kidney tumor segmentation in ct imaging., 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Qixin Hu, Yixiong Chen, Junfci Xiao, Shuwen Sun, Jienong Chen, Alan L. Yuille, and Zongwei Zhou. Label- free liver tumor segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7422 7432, June 2023.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Kaikai Huang, Antonio Tejero-de Pablos, Hiroaki Yamane, Yusuke Kurose, Junichi Iho, Youji Tokunaga, Makoto Horie, Keisuke Nishizawa, Yusaku Hayashi, Yasushi Koyama, et al. Coronary wall segmentation in ceta scans via a hybrid net with contours regularization. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 1743-1747. IEEE, 2020.
Fabian Isensec, Jens Petersen, Andre Klein, David Zimmerer, Paul F Jaeger, Simon Kohl, Jakob Wassorthal, Gregor Koehler, Tobias Norajitra, Sebastian Wirkert, et al. muu-net: Self-adapting framework for u-net- based medical image segmentation. arXic preprint 1XXi:1889.114488 2018.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference computer vision, pp. 709-727. Springer, 2022.
Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, and Dorit Merhof. Diffusion models medical imaging: comprchensive survey. Medical image analysis, 88:102846, 2023.
Andrew Lin, Nipun Manral, Priscilla McElhinney, Aditya Killekar, Hidenari Matsumoto, Jacek Kwiecinski, Konrad Pieszko, Aryabod Razipour, Kajetan Grodecki, Caroline Park, et al. Deep learning-enan enabled coro- nary ct angiography for plaque and stenosis quantification and cardiac risk prediction: an international multicentre study. The Lancet Digital Health, 4(4):c256-c265, 2022.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys, 55(9):1-355 35, 2023.
Riccardo Marano, Giuseppe Rovere, Giancarlo Savino, Francesco Ciriaco Flammia, Maria Rachele Pia Carafa, Lorenzo Steri, Biagio Merlino, and Luigi Natale. Ccta in the diagnosis of coronary artery disease. La radiologia medica, 125:1102-113,, 2020.
George A Mensah, Valentin Fuster, Christopher JL Murray, Gregory A Roth, Global Burden of Cardiovas- cular Diseases, and Risks Collaborators. Global burden of cardiovascular diseases and risks, 1990-2022. Journal of the Americon College of Cardiology, (2(25))2350-2477 2023.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot dustt--disttibuution detection via prompt learning. Advances in Neural Information Processing Systems, 36:76298-76311 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMIR, 2021.
Mehran Pesteie, Purang Abolmacsumi, and Robert N Rohling. Adaptive augmentation of medical data using independently conditional variational auto-encoders. IEEE transactions on medical imaging, 38(12): 2807-2820, 2019.
Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Khan. Promptir: Prompting for all-in-one image restoration. In seirth-seventh Conference on Neural Information Processing Systems, 2023.
Chongyu Qu, Tiezheng Zhang, Hualin Qiao, Jie Liu, Yucheng Tang, Alan Yuille, and Zongwei Zhou. Annotating 8,000 abdominal et volumes for multi-organ segmentation in three weeks. arXie preprint arXiv:230.006666 3, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pp. 2021.
PmLR,
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assiste InterventionMIICCA. 2015: 18th inter- national conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Yizhe Ruan, Lin Gu, Yusuke Kurose, Junichi Iho, Youji Tokunaga, Makoto Horie, Yusaku Hayashi, Keisuke Nishizawa, Yasushi Koyama, and Tatsuya Harada. Physiology-aware polysnake for coronary vessel seg- mentation. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pp. 8855-8864, February 2025.
Patrick W Serruys, Hironori Hara, Scot Garg, Hideyuki Kawashima, Bjarne L Norgaard, Marc R Dweck, Jeroen J Bax, Juhani Knuuti, Koen Nieman, Jonathon A Leipsie, et al. Coronary computed tomographic angiography for complete assessment of coronary artery disease: Jacc state-of-the-art review. Journal of the American College of Cardiology, 78(7):713-77) 2021.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. ar arXiv preprint or arvv::2010.155 15980, 2020.
Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Anabysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML- CDS 2017, Held in Conjunction with MICCAI 2017, Québec Cily, QC, Canada, Seplember 14, Proceedings 3, pp. 240 248. Springer, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Linshan Wu, Jiaxin Zhuang, Xuefeng Ni, and Hao Chen. Freetumor: Advance tumor segmentation via large-scale tumor synthesis. arXiv preprint arXiv:2406.0122 0124, 2024.
Yuan Xue, Jiarong Ye, Qianying Zhou, Rodney Long, Samoer Antani, Zhiyun Xue, Carl Cornwell, Richard Zaino, Keith C Cheng, and Xiaolei Huang. Selective syuthetic augmentation with histogan for improved histopathology image classification. Medicol image analysis, 67:101816, 2021.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847,
2023.
Zheyuan Zhang, Lanhong Yao, Bin Wang, Debesh Jha, Gorkem Durak, Elif Keles, Alpay Medetalibeyoglu, and Ulas Bagci. Diffboost: Enhancing medical image segmentation via text-guided diffusion model. IEEE Transactions on Medical Imaging, 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasionnla models. International Journal of Computer Vision, 130(9)22337-2348, 2022.
16


Reviews:
[{"review_id": "IUl4RHMtI3", "reviewer": "Reviewer_j7ka", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The central concept of using learnable, class-specific prompts to guide a diffusion model for multi-class medical data synthesis is novel and well-motivated. It directly addresses the key challenges of data scarcity and severe class imbalance in a scalable way, without needing to train separate models for each lesion type.\n\n- The method shows consistent and significant improvements across different anatomies (CCTA, kidney, liver, pancreas), different lesion types, and different segmentation backbones (U-Net, nnU-Net). The large improvement in Dice Score for the difficult \"non-calcified plaque\" class (e.g., from 23.1 to 44.3 for nnU-Net) is particularly compelling.\n\n- The paper is well-written and easy to follow. The methodology is explained clearly, and Figure 1 provides an excellent high-level overview of the entire pipeline. Figure 2 effectively illustrates the more complex transformer-based prompt mechanism.\n\nWeaknesses:\n- The primary motivation and validation for CCTA plaque segmentation are done on a private dataset. While common in the medical field due to privacy concerns, this may not be favorable for reproducibility. \n\n- Showing a few examples of the generated CCTA and tumor images (along with their synthetic masks) would help readers visually assess the anatomical realism and diversity that the paper claims.\n\n- A brief discussion of the training time for PromptLesion and the inference time to synthesize a new batch of images would be valuable context for researchers or clinicians looking to adopt this method.\n\n- Does the proposed augmentation method have a positive impact on the transformer-backbone related methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose PromptLesion, a framework for medical data augmentation that uses a prompt-conditioned latent diffusion model. The main contribution is the use of learnable, lesion-specific prompts to guide the synthesis process, enabling a single generative model to produce diverse and anatomically realistic examples for multiple classes (e.g., calcified vs. non-calcified plaques, or tumors in different organs), thereby addressing data scarcity and severe class imbalance. They further enhance this by introducing an implicit, transformer-based prompting mechanism that outperforms simple class conditioning. Through extensive experiments on both private CCTA and public multi-organ tumor datasets, they demonstrate that augmenting training data with these synthetic lesions significantly improves the performance of downstream segmentation models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please try to address the weakness or give some discussion.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bkcBMWkhmC", "reviewer": "Reviewer_Jr97", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. An interesting adaptation of current diffusion generation techniques to a practical use case (medical image segmentation).\n2. Clearly written paper, easy to follow. Approach is straight forward.\n3. Comprehensive experiments to evaluate the proposed method.\n\n\n**Weaknesses**\n1. Limited CCTA test set (10 instances) of a private dataset.\n2. Missing comparison with recent conditional diffusion models, evaluating segmentation performance using their generated images.\n3. Last paragraph of section 3.2.1 mention, “we found that inserting generated synthetic lesions into authentic anatomical backgrounds substantially improved realism and segmentation performance compared to purely synthetic images”. But couldn’t find the evidence to support this under results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores a prompt based conditioning diffusion generation to generate anatomically and pathologically diverse synthetic images for medical image segmentation to address scarcity of annotations and class imbalance. From the primary target of CAD data, the authors extend the approach to handle multi-organ tumour synthesis, reporting higher segmentation performance than compared baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I suggest including a test on public CCTA dataset to improve the reproducibility, if possible. Although public tumour datasets are used, the key claim of plaque segmentation improvements cannot be verified independently.\n\n2. The paper claims an architectural advancement (the first diffusion-based framework integrating lesion specific prompts for conditioning), hence, while understanding the computational demand, I suggest including a comparison with at least 1–2 related conditional diffusion models, evaluating segmentation performance using their synthetic data.\n\n3. Please point to the relevant results (point 3 under weaknesses).\n\n4. Please clarify how DiffTumor was trained for ablation 4.5.1 so that the difference between its values reported for Table 1, 2 vs ablation 4.5.1 is clearer.\n\n5. Please clarify the early stopping condition or metric on validation set.\n\n6. In figure 1 step1 (Autoencoder Model), what is the importance of having a Transformer.? Because PromptLesion does not address generating long sequences in quantization.\n\n7. In general, I suggest simplifying figure 1 to only show relevant parts for each step (i.e: step 3 is named segmentation model but the whole pipeline is shown). \n\n8. Please clarify if $z^{helthy}$ is the dot product between non-lesion pixels of the mask and already quantised latent code $z$ (section 3.2 first paragraph). If so, what is the intuition behind getting the dot product between already quantised input and features in input mask.?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QosvCTyYF4", "reviewer": "Reviewer_waki", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper addresses a real clinical challenge in assessing coronary artery disease, where non-calcified plaques are difficult to segment due to severe class imbalance and low contrast with surrounding tissues.\n- Unlike existing methods like DiffTumor, which require separate models per lesion type, PromptLesion synthesizes multiple lesion classes within a single model through prompt conditioning, improving efficiency and scalability.\n- This paper demonstrates substantial improvements across all tasks, including Non-calcified plaque, consistent gains across multiple organs and architectures, and outperforms both single-class and GAN-based baselines.\n- This paper validates two distinct clinical applications (CCTA plaques and multi-organ tumors) and demonstrates generalizability beyond a single anatomical region.\n- Ablation studies provide meaningful comparisons of explicit (one-hot) vs. implicit (transformer-based) prompts, showing consistent improvements with implicit prompts.\n\nWeaknesses\n- The approach primarily combines existing components (VQGAN from DiffTumor, prompt conditioning inspired by PromptIR, and standard diffusion models) without significant methodological innovation.\n- The CCTA dataset is private, limiting the reproducibility of key results.\n- No computational cost or inference time analysis.\n- Lacks discussion of what prompts actually learn or how they influence the generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a prompt-conditioned diffusion framework that generates multiple lesion types within a single model. The paper addresses the issues of data scarcity and class imbalance in medical image segmentation. The key contribution is the use of lesion-specific prompts to guide the diffusion-based synthesis. The proposed method employs either one-hot vectors or transformer-based embeddings. Transformer-based embeddings achieve 2-3% better performance. Validated on CCTA plaque segmentation and multi-organ tumor tasks, the method demonstrates substantial improvements, particularly for underrepresented classes (e.g., non-calcified plaque DSC: 21.3% -> 39.0%), outperforming single-class augmentation baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide complete architecture details for the transformer-based prompt encoder (layer dimensions, attention heads, etc.).\n- Include all training hyperparameters (learning rates, batch sizes, diffusion steps, noise schedules).\n- Specify computational requirements (GPU memory, training time, inference time).\n- Add implementation details for prompt injection into the U-Net at multiple scales.\n- Add perceptual quality metrics (FID, LPIPS) to assess the realism of synthetic lesions.", "broader_impact_concerns": "The paper lacks a Broader Impact Statement, which is essential for medical AI research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)