LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models
Authors: Hantao Zhang
Yuhe Liu
Jiancheng Yang
Shouhong Wan
Xinyuan Wang
Wei Peng
Pascal Fua
Venue: ICLR 2025 Spotlight (2025)
Keywords: data synthesis
diffusion models
cardiac MRI
lung nodule CT
segmentation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose LeFusion, a lesion-focused diffusion model that synthesizes diverse lesion image-mask pairs from lesion-free images, enabling controllable multi-peak and multi-class lesion generation, significantly improving segmentation models.

Abstract: Patient data from real-world clinical practice often suffers from data scarcity and long-tail imbalances, leading to biased outcomes or algorithmic unfairness. This study addresses these challenges by generating lesion-containing image-segmentation pairs from lesion-free images. Previous efforts in medical imaging synthesis have struggled with separating lesion information from background, resulting in low-quality backgrounds and limited control over the synthetic output. Inspired by diffusion-based image inpainting, we propose LeFusion, a lesion-focused diffusion model. By redesigning the diffusion learning objectives to focus on lesion areas, we simplify the learning process and improve control over the output while preserving high-fidelity backgrounds by integrating forward-diffused background contexts into the reverse diffusion process. Additionally, we tackle two major challenges in lesion texture synthesis: 1) multi-peak and 2) multi-class lesions. We introduce two effective strategies: histogram-based texture control and multi-channel decomposition, enabling the controlled generation of high-quality lesions in difficult scenarios. Furthermore, we incorporate lesion mask diffusion, allowing control over lesion size, location, and boundary, thus increasing lesion diversity. Validated on 3D cardiac lesion MRI and lung nodule CT datasets, LeFusion-generated data significantly improves the performance of state-of-the-art segmentation models, including nnUNet and SwinUNETR.

Published as a conference paper at ICLR 2025
LEFUSION: CONTROLLABLE PATHOLOGY SYNTHESIS VIA LESION-FOCUSED DIFFUSION MODELS
Hantao Zhang 12; Yuhe Liu?, Jiancheng Yang';
Shouhong Wan",
Xinyuan Wang". Wei Peng", Pascal Fua'
1 Swiss Federal Institute of Technology Lausanne (EPFL), Switzerland UUniversity of Science and Technology of China (USTC), China
BBeihang University, China SStanford University, USA
ABSTRACT
Patient data from real-world clinical practice often suffers from data scarcity and long-tail imbalances, leading to biased outcomes algorithmic unfairness. This study addresses these challenges by generating lesion-containing image- segmentation pairs from lesion-free images. Previous efforts in medical imag- ing synthesis have struggled with separating lesion information from background, resulting in low-quality backgrounds and limited control over the synthetic out- put. Inspired by diffusion-basel image inpainting, we propose LeFusion, a lesion- focused diffusion model. By redesigning the diffusion learning objectives to fo- cus on lesion areas, we simplify the learning process and improve control over the output while preserving high-fidelity backgrounds by integrating forward- diffused background contexts into the reverse diffusion process. Additionally. we tackle two major challenges in lesion texture synthesis: multi peak and multi-class lesions. We introduce two effective strategies: histogram-base based tex- ture control and multi-channel decomposition, enabling the controlled generation of high-quality lesions in difficult scenarios. Furthermore, we incorporate lesion mask diffusion, allowing control over lesion size, location, and boundary, thus increasing lesion diversity. Validated 3D cardiac lesion MRI and lung nodule CT datasets, LeFusion -egnerreeee data significantly improves the performance state-of-the-art segmentation models, including nnUNet and SWinUNETR. Code and model are available at
INTRODUCTION
The development of AI for healthcare often suffers from data scarcity (lbrahim et al. 202 1 Schäfer et 2004). In most biomedical scenarios, the number of pathological subjects is significantly lower than that of normal ones. This discrepancy primarily arises from the naturally occurring distribution of patient data, which frequently exhibits long-tail characteristics (Yang et al.) 2022, Zhang et al.) 2023). Additionally, potential biases in data collection can introduce issues related to algorithmic fairness (Xu et al. [022) Chen et al.) 2023 Yang et al.) (2024), as well as concerns about security and privacy (Price Cohen 2019; Qayyum et al. (2020). As result, has been argued that "synthetic
data can be better than real data" (Savage 2023).
Generative lesion synthesis is promising approach generating diverse medical data, benefiting many medical applications (Khader et al. 2023). By learning from lesion-containing data, gen- erative models can synthesize various types of lesions, which in turn benefit downstream applica- tions (Han et al.) 2019 Jin et al.) 2021 Yang et al. 2019 Lyu et al. 2022a Shin et al. 2018 Pishva et al. 2023 Du al. 20023 Lyu et al. 2022b). While a range of generative methods have been explored, they often struggle to preserve high-quality backgrounds outside of the lesion ar- eas. This is because generating anatomically correct backgrounds in the human body is far more challenging than synthesizing isolated lesions. Moreover, these methods often lack control over key
"This work was conducted during the first author's research internship at EPFL. *oorrsponning author: Jiancheng Yang (j (jiancheng. bochheng.yaggee yynggepp11 (ch),, who led the project.

---

Published as a conference paper at ICLR 2025
Lung Nodules
Myocardisl Damage
Foreround Mask
Conditionll Reverse Diflusion
Leace- Recooel
Backgrood
(a) Standard Conditional Diffusion
Lung Nodules
Myocordial Damage of
Forward
Diflusion
Nooe
Reverse
Fonggrand Nusk
Diflusion
(b) LeFusion: Lesion-Focused Diffusion
Lung Nodules v
Myocartial Damaae V
Histogram-BB Baasddd Textura
Muti-Charnnl
Lesion Mask Diflusion
Decomposition
(c) LeFuiion
Textures
Figure 1: Standard Conditional Diffussin vs. Lesion-Focused Diffusion (LeFusion). (a) Stan- dard Conditional Diffusion concatenates background, lesion mask and noise as inputs and outputs both lesion and background, risking background integrity and wasting capacity on difficult but un- necessary background generation, especially in data-limited settings. (b) LeFusion uses forward- diffused backgrounds and reverse-diffused foregrounds as input. The model generates only the lesion, ensuring realistic background preservation and simplifying the task. (c) LeFusion with Fine Control of Lesion Textures and Masks introduces histogram- texture control for multi-peak le- sions (lung nodules), multi-channel decomposition for multi-class lesions (myocardial damage), and lesion mask diffusion for control over size, location and boundary, enhancing quality and diversity.
aspects of lesion generation, including texture type, size, location, and mask alignment. These issues can severely degrade the performance of downstream applications, such as segmentation algorithms trained using such synthetic data. Fig. (a) illustrates standard diffusion models as an example.
One approach to avoiding the complexity of background generation is to start from readily available normal scans and synthesize lesions into them. This involves generating lesion masks and filling them with appropriate textures, ensuring perfect background preservation and mask alignment, while also allowing precise control over lesion size and location. this paper, we refer to this method as backgrounn-ppeeeerving lesion synthesis. This approach has led to a resurgence of hand-crafted methods, which have been used to model COVID-19 lesions (Yao et al. (022)) and liver tumors (Hu al. 2023). However, these methods rely on heuristics that do not generalize well.
Inspired by diffusion-based image inpainting schemes (Lugmayr et al. 2022 Avrahami et al, 2022), it has been shown that explicitly integrating real background context during the diffusion process ensures realistic background preservation outside lesion masks. Rather than using the background conditional inputs (Rombach et al. 2022). these methods directly incorporate forward-diff diffusedd background contexts into the reverse diffusion process. However, these approaches are training free and do not focus on specific inpainted content. In contrast, our study emphasizes lesion generation. In data-limited scenarios, it is more efficient for the model to focus solely on lesion synthesis.
To this end, we propose LeFusion, lesion-focused diffusion model (Fig. I and Fig. [b.. We design the diffusion learning objectives to focus solely on lesion data. Similar to diffusion-based inpainting. the input combines forward-diffused backgrounds with reverse-diffused foregrounds, while the model reconstructs only the lesion, avoiding the need to allocate capacity to learning com- plex backgrounds. Furthermore, we address two major unresolved challenges lesion synthesis: 1) multi-peak lesions, where lesions have distinct types, and 2) multi-class lesions, where multiple classes of lesions need to generated simultancously. For the multi-peak challenge, we introduce histogram-based texture control, integrating lesion texture histograms during training as condi- tion, which allows control over lesion types during inference. We find that explicitly controlling the histogram is crucial when generating lesions on normal scans; otherwise, the model tends to produce lesions biased toward healthy appearances. Notably, this histogram- based control method generic and does not require any additional information beyond image-mask pairs. To handle the
2

---

Published as a conference paper at ICLR 2025
multi-class challenge, we propose strategy for joint modeling of multi-class lesions through multi- channel decomposition, where the diffusion model generates different lesion classes via separate channels and then combines them into single image. Finally, we introduce lesion mask diffusion, enabling control over size, location and boundary, thereby increasing the diversity of lesion masks.
We validate LeFusion 3D lung nodule CT datasets (Armato III et al. 2011) and cardiac lesion MRI (Lalande et al.] 2022), demonstrating its effectiveness in addressing both the multi pelti-peak and multi-class challenges while generating high-quality synthetic lesions. In downstream segmentation tasks, we show that LeFusion-generated data significantly enhances the performance of state-of-the- art models such as nnUNet (Isensee al. 2021) and SwinUNETR (Hatamizadeh et al. 2021..
RELATED WORK
2.1 GENERATIVE MODELS FOR LESION SYNTHESIS
Lesion synthesis using generative models has garnered significant attention for its potential to cre- ate diverse and realistic medical datasets, particularly in addressing the scarcity and imbalance of pathological data in biomedical applications. Early approaches have employed variational autoen- coders (Kingaai & Welling. 2013), generative adversarial networks
et al.) 2020 and more recently, diffusion models (Ho et al.) 2020) to generate synthetic lesions across various medical imaging modalities and applications, including lung nodules (Han et al.)
2019 Jin et al. 2021 Yang al. 2019) and COVID-19 lesions (Lyu et al. 2022a) in CT scans, colon polyps in colonoscopy (Shin et al. 2018; Pishva et al.) [2023] Du et al., 2023). tumor cells in microscopy (Horvath et al. 2022), brain and brain lesions in MRI (Billot et al. (023), diabetic
lesions in retinal images (Wang et al. 2022), and synthetic liver tumors (Lyu et al. 20226)
However, persistent challenge for these methods is preserving anatomically accurate backgrounds alongside the lesions. In medical imaging, the background must respect the anatomical structure of the human body, which makes generating realistic backgrounds significantly more difficult than synthesizing isolated lesions. particularly in large-scale 3D images. While recent studies (Hamamci al., 2024; Peng al.) (024] have begun to address large-scale 3D medical image generation, these methods often require sigmificant computational resources and extensive data.
Another limitation of current methods is the lack of explicit control when generating image- pairs. Typically, both the lesion and its corresponding mask are generated simultaneously, without explicit constraints linking the two. This results in limited control over key lesion properties, such as texture, size, location, and alignment between the image and mask. The absence of high-qualit background preservation and fine control over these properties hinders the scalability and effective- ness, negatively impacting the performance of downstream tasks such as segmentation.
2.2 BACKGROUND- PRESERVING LESION SYNTHESIS
In clinical practice, normal scans (either whole or partial) are far more abundant than pathological ones. For instance, in lung nodule cases, most pathological scans contain only single lesion, yet traditional lesion synthesis methods often focus on small crops around the lesion, utilizing as little as < 1%o of the original data and leaving large portions of the normal background unused. This raises the question of whether it is necessary to rely on deep models to generate normal backgrounds.
Background-presse preservvng lesion synthesis addresses this issue by starting from normal scans and syn- thesizing lesions through filling textures into manually generated lesion masks. This approach sepa- rates the generation of lesion masks and textures, allowing for finer control over lesions while main- taining the original background structure. Prior work has predominantly relied on heuristic-based methods (Yao et al. 2021| Hu et al., 2023), leveraging the abundance normal backgrounds to gen- erate lesions of varying sizes and textures at different locations. These studies have demonstrated that the generated data can significantly benefit downstream segmentation tasks.
However, these hand-crafted approaches rely heavily on manual adjustments to ensure that the gen- crated lesions resemble real-world pathology. For example, Hu et al. (2023) manually set grayscale values for texture synthesis, and lesion shapes are generated using morphological operations from
For example, a 64 cube crop from 512 volume occupics < 0.2% of the total voxels.
3

---

Published as a conference paper at ICLR 2025
ellipsoidal masks. Such hand-crafted rules limits the scalability and generalizability of these meth- ods. In more complex scenarios, such as multi-peake and texture-rich lung nodules or multi-class cardiac lesions, these methods tend to fail (see S.c This highlights the need for more flexible and robust data-driven techniques to address these challenges.
A recent study (Chen et al. 2024 uses conditional diffusion (Ho et al.. 2020 Rombach et al.) 2022) lesion synthesis. The image is first encoded with VQGAN (Esser et al. 2021), and latent diffusion (Rom- bach et al.) 2022) learns both the background and lesion, using the lesion mask and background (excluding the lesion) as conditional inputs. However, as the model still needs to generate back- ground, the preservation of background integrity cannot be theoretically guaranteed. While it is possible to fill the area outside the mask with real background data, this may lead to inconsistencies in the final output. From our findings, this approach struggles with high-quality background gener- ation, affecting downstream applications (Sec.4.2), particularly in data-limited scenarios. We also tested similar image-space conditional diffusion model, which showed similar limitations, though image-space diffusion outperformed latent diffusion due to constraints imposed by the autoencoder.
to synthesize abdominal tumors, as illustrated in Fig. Th. building on background-preservee
A few concurrent studies (Lai et al. 2024 Wu et al. 2024 Zhu et al. 2024) have employed advanced generative models for lesion synthesis. Due to differences in research focus or/and the unavailabil- ity of their code/models, comprehensive comparison could not be conducted. Besides, while some studies leverage inpainting techniques to model specific regions of interest (Rouzrokh et al., 2022, Hansen al. 2024), these methods cannot theoretically guarantee background preservation. Our work emphasizes lesion-specific synthesis rather than broader generative model paradigms, focus- ing primarily on diffusion-based approaches within the framework of background-ppesseriin lesion synthesis. Comparisons with other generative model paradigms were not included in this study.
LEFUSION: LESION-FOCUSED DIFFUSION MODEL
We propose LeFusion, a lesion-focused diffusion model that concentrates solely on lesion. In Sec by combining forward-diffused backgrounds with reverse-diffused foregrounds, LeFusion reconstructs only the lesion, eliminating the need to model complex backgrounds. To address key challenges in lesion texture synthesis (Sec LeFusion introduces histogram- based texture control for multi-peak lesions, allowing control over distinct lesion types, and a multi-channel decomposi- tion strategy for multi-class lesions, where classes are generated in separate channels and combined into a single image. In Sec. 3.3 lesion mask diffusion enables control over lesion size, location and boundary, increasing mask diversity and enhancing lesion synthesis quality and flexibility.
3.1
GENERATION VIA INPAINTING
Decoupled Lesion and Background Generation. Inspired by diffusion-based inpainting (Lug-) mayr et al. 2022 Avrahami et al., 2022), we aim to decouple the generation of lesions and back- ground. As shown Fig. 2 inpainting predicts the missing parts of an image, particularly lesions. For standard diffusion models (Sohl-
et al. 2015; Ho et al. 2020), the inference process starts by sampling a noise vector 2T ~ (O, 1) and gradually denoising to produce the output image Ti- We focus on 3D images only in this study, while the method can be easily extended to 2D. The original grayscale image is denoted as dtp E Dx xxx1 where D, H, and W represent the 3D image size. The reverse diffusion step from Ie to Iz-1 is decoupled into lesion and back- ground components, as shown below. Here, My and Mn represent the lesion foreground mask and background mask. The lesion 0 is predicted through the reverse diffusion process pe using a 3D U-Net, while - 1 is derived from to by adding noise through forward diffusion q. āe defined as [1 - B.). 3r represents the variance schedule parameter, which determines the rate which noise is gradually injected over time.
= N(o0-1; (o t), Ea ()),
( = Nat;: Vario. (1 II)
(2)
8--1 = 0 2 M, + 2z-1 ② Mp.
(3)
This approach preserves the background accurately without requiring prediction.
4

---

Published as a conference paper at ICLR 2025
Training
Forward Diflusion
Lesion Focused
A Jhathat on Noxoed
Cuntrol Halopamm
My DhdttWWan Leeion Foroground Mask
ExtEx an Dena aed
Uai so
RiSESP st image
Inference
For oma
Forward Diffusion g
Lesion-Focused
Reverse Diflusion
Ri-1
9--1
Bi-a Denclsed
Inibal Naza
Donoised
Forward Officedd Impudlad
Background
Luaizn Ohrpul
Foreground Mask
Maak
Outpuii
Figure 2: LeFusion: Lesion-Focusee Diffusion Model. The top illustrates the training process of LeFusion, while the bottom shows the inference. During training, LeFusion avoids learning unnec- essary background generation using a lesion-focused loss. In inference, by combining forward- diffused real backgrounds with reverse-diffused generated foregrounds, LeFusion ensures high- quality background generation. Additionally, we introduce histogram-basee texture control to han- dle multi-peat lesions and multi-channel decomposition for multi-class lesions.
Making Diffusion Model Lesion-Focused. The method above, while suitable for all diffusion- based inpainting, cannot guarantee that the denoised output 0--1 is focused on the lesion area. Unlike standard diffusion training, where the target region may vary, know that the part to be inpainted is specifically the lesion in our application. Therefore, we can design the model to predict only the lesion and ignore other regions. To achieve this, introduce lesion-focused loss during training. The general diffusion process starts with the original image fo. adding Gaussian noise over T' time steps (forward diffusion). The neural network is trained predict the noise distribution at time step (reverse diffusion). conditioned on the noised image i; and the time step. To ensure the model focuses only on the lesion, mask M is applied, calculating the loss exclusively within the lesion region. The training objective is defined as follows, where E xxxxxxXX represents the noise sampled from Gaussian distribution:
Bgo.c~A((, Mylle pe (fe, [12]
(4)
Despite the change in the training objective, inpainting inference remains unaffected. As shown in Eq. 3 outside My, the predicted lesion ot-1 is replaced by the real noised background Lt-1.
3.2 FINE CONTROL OF LESION TEXTURES
Handling Multi-Peak Distributed Lesions. In the section above, the model relies on the noised back- ground 2t to infer the texture of le- sion within mask. While this works for lesions with minimal tex- ture differences (as with cardiac sions), it becomes problematic with multi-peak data. As shown in Fig. lung nodules exhibit distinct texture types. We empirically show that rely- ing solely on the background can lead to mode collapse.
Figure 3: Illustration of Lung Nodule Texture Histogram Distribution. Samples are clustered into three groups based the grayscale image histogram of lesions. The visual- ized differences between groups are significant, indicating a typical multi-peak distribution. These clusters roughly respond to ground-glass, part-solid, and solid nodules.
To address this, propose simple yet effective approach, histogram based texture control. The lesion texture histogram h is used as a condition via cross attention Rombach et al. (2022), i.e.,
0:-1 ~ Po h,
During training, the histogram is computed from the ground truth, and during inference. texture types can be controlled by adjusting the histogram. Notably, this approach requires no additional lesion type annotations, such nodule attenuation.

---

Published as a conference paper at ICLR 2025
lterations
Procassing
Inilial Noise
Control Sphere
Denoised Oulput
Boundary Mask
Masked Denoised Oulput
Figure 4: DiffMask: Lesion Mask Diffusion. To achieve fine control over lesion size, location, and boundary, we propose two key designs: the boundary mask and the control sphere. The boundary mask removes areas outside the boundary at each diffusion step. The control sphere, trained using the bounding spheres of real masks, enables control over size and location during inference.
In Sec. 4 we demonstrate that the proposed histogram- -based texture control is crucial for generating lesions on normal scans. Without it, models tend to fail, biasing towards healthy appearances and producing overly subtle lesions, which degrades the performance of downstream segmentation tasks using these synthetic data.
Joint Modeling of Multi-Class Lesions. The method above focuses single-class lesions, but many medical applications, such as cardiac MRI, require modeling multiple lesion types, like my- ocardial infarction (MI) and persistent microvascular obstruction (PMO). To capture correlations between lesion types and generate textures for multiple lesions simultaneously, we use joint mod- eling strategy called multi-channel decomposition, where each channel corresponds different lesion type. The diffusion model generates each lesion in its respective channel, and they are com- bined through lesion masks.
We expand the input image on E RD to i, E DDXXxXXx based on the number of lesion classes n, where n in the cardiac MRI experiments. Similarly, e E Dxxxxxxx and 0t-1 Pe t) E Dxxxxxxn are extended to n channels. The training objective is:
E eog.eo
Σ
M ||e - pe (&e, (11)))
(6)
where (1) refers to the channel i of *, To combine these channels, compute Ex
M 0
3.3 DIVERSIFYING LESION MASKS VIA DIFFMASK
To further enhance the controllability and diversity of lesion generation, we introduce lesion mask diffusion (DiffMask). As shown in Fig. 4 to achieve fine control over lesion size, location, and boundary, we propose two key designs: the boundary mask and the control sphere. The former removes areas outside the boundary at each diffusion step, ensuring the generated mask stays within reasonable limits, while the latter manages the size and location of the lesion. During training, the control sphere is the bounding sphere of a real mask and is concatenated as a condition to the DiffMask input, with the real mask serving the target. In inference, and location to generate the desired lesion masks. terms of implementation, the architecture of DiffMask is similar to the texture diffusion model, also employing multi-channel decomposition to capture shape correlations and spatial distributions between multiple lesion masks. Each output channel is responsible for generating the lesion shape mask of specific lesion. Finally, apply smoothing kernel as a post-processing step.
can adjust the size
4 EXPERIMENTS
4.1 SETUP
Dataset. LIDC: Multi-Peak Lung Nodule CT. We use LIDC dataset (Armato III et al. 2011), which contains 1,010 chest CT scans, from which 2,624 regions of pathological (P) interest (ROIs) cor- responding to lung nodules were extracted, along with 135 cases of healthy patients. The dataset was divided into 808-case training set, comprising 2,104 lung nodule ROIs, and a 202-case test set, containing 520 lung nodule ROIs. Additionally, 3,076 normal N) ROIS were cropped from the 135 healthy patients, representing regions where lung nodules typically appear. These normal

---

Published as a conference paper at ICLR 2025
Table 1: Downstream Lung Nodule Segmentation Dice( (1) and NSD(%) (1) on LIDC (2011). P: real pathological cases. P*/N": synthetic pathological cases from pathologi- cal/normal cases. N" more synthetic data than N'. Bold numbers indicate the best, with red highlighting significantly adverse effects (> 1% lower) compared to the baseline, and blue indi- cating significantly positive effects (> 1% higher).
nnU-Net 2021
SwinUNETR (202
Methods
Training Setting
Dice (t)
NSD (t)
Dice (1)
NSD
Baseline
78.26
88.90
78.38
88.67
Texture Synthesis with Real Masks p
Hand-Crafted (0233)
P+P'
76.80
87.94
76.11
86.31
Sond-Diffusion 2020 2022
P+P
77.05
87.69
77.51
88,09
Sond-Diffusion (L) (2024)
P+P"
76.66
87.2
76.44
86,56
RePaint (2022)
P+P"
77.57
88.07
77.14
87.96
LeFusion (Ours)
P+P'
78.77
89.25
78.43
88.54
LeFusion-H (Ours)
P+P
80.62
90.90
80.95
90.98
Texture Synthesis with Hand-Crafted Synthetic Masks
(023) on
Hand-Crafted (2023)
P+N
75.10
85.50
74.88
84.64
Cond-Diffusion Diffusion 2020 2022
P+N
76.62
86.44
76.66
87,20
Sond-Diffusion (L) 2024
P+N"
76.71
86.83
77.20
87,88
LeFusion (Ours)
P+N
77.67
87.94
77.98
88.42
LeFusion-H (Ours)
P+N
80.19
89.75
80.08
90.42
Texture Synthesis with Copied Masks
2023) on N'
Copy-Paste
P+N'
77.29
87.60
77.80
88.84
Hand-Crafted 2023
P+N"
76.04
86.57
76.58
87.72
Cond-Diffusion 2020
2022
P+N"
77.00
87.68
87.40
Cond-Diffssion (L) 2024
P+N
77.15
87.83
77.38
87.51
LeFusion (Ours)
P+N"
78.49
89.22
78.55
89.06
LeFusion-H (Ours)
P+N
81.11
91.77
81.10
91.67
Enhanced with Diffusion Based Synthetic Mask (DiffMask)
LeFusion- H+DiffMask (Ours)
P+N
82,66
92.49
82.63
92.77
LeFusion-H- H-DiffMask (Ours)
P4N
83.19
93.21
833.07
93.10
I LcFusion-H-DifMask (Ours)
P+P'+N"
83.44
93.35
83.13
93.20
ROIs were used for data augmentation in the experiments. Emidec: Multi- Clatis Cardiac Lesion MRI. The Emidec dataset (Lalande et al.) (2022) consists of examinations featuring DE-MRI in short-axis orientation. This dataset offers access to 100 labeled cases, including 33 normal (N) and 67 pathological (P). The annotations cover classes: background, left ventricle (LV), myocardium (Myo), myocardial infarction (MI), and persistent microvascular obstruction (PMO). We split the 67 P cases into 57 for training and 10 for testing. The 57 P cases used to train the data synthesis model. In the downstream evaluation (Sec. 4.2) we use those models to synthesize cases based on both 57 P and 33 N as the training set.
Method Comparison. The following synthesis algorithms are compared with the LeFusion.
Copy-Paste. We used the masks from real lesion data and matched them with normal data, copying original lesion textures onto normal cases to generate new synthetic data.
Hand-Crafted (Hu el al. 2023). The lesion mask is represented by the overlapping of multiple ellipsoidal lesion masks, followed by several random morphological operations. The texture is ap- proximated using Gaussian noise and softened through interpolation and Gaussian filtering.
RePaint (ugmaayr et al. 2022 or Blended- Sinidded-iffusiionnno Avrahami et al. (002)). These methods are standard image diffusion models during training, while in inference, they remove and re-fill in the texture within the lesion mask, by combining forward-difuseed backgrounds with foregrounds. The model employs global training loss, which lacks the capability to focus lesion information. Due to the absence guidance from lesion category information, is unable to specify corresponding lesions and is confined to simulating the generation of single-class lesions;

---

Published as a conference paper at ICLR 2025
Table 2: Downstream Cardiac Lesion Segmentation Dice( %) (f) on Emidec (2022). The NSD
metric is provided in Tab. [T MI and PMO are [lesion classes. P: real pathological cases. P'/N: synthetic pathological cases from pathological/normal cases. N more synthetic data than N'. Bold
numbers indicate the best, with red highlighting significantly adverse effects (> 1% lower) com-
pared to the baseline, and blue indicating significantly positive effects (> 1% higher)
Training Setting
nnU-Net (021
SwiaUNETR 2021 MI Dice (t) PMO Dice (1)
Methods
MI Dice (t)
PMO Dice (1)
Baseline
P
68.61
36.32
57.79
35.76
Texture Synthesis with Real Masks p
Hand-Crafted (0233)
P+P'
69.60
36.06
57.64
34.96
Cond-Diffusion Diffusion 2020 2022
P+P
66.89
37.76
56.75
36.31
Cond- Minf-Diffusion (L) (2024)
P+P"
68.07
31.93
56,97
32.72
RePaint (2022)
P+P
69.14
28.93
55.14
33.86
LeFusion-S (Ours)
P+P"
69.88
34.79
57.85
35.63
LeFusion-J (Ours)
P+P
69.95
38.01
59.61
37.99
Texture Synthesis with Hand-Crafted Synthetic Masks (2023) on
Hand-Crafted (2023)
P+N'
68.19
35.73
56.18
35.01
Mond-Diffssion 2020 2022
P+N"
67.41
31.03
56.73
35,28
Cond-Diffin Diffusion (L) (0244
P+N
67.08
36.31
56.70
33.84
LeFusion-S (Ours)
P+N' 69.17
37.18
59.42
34.83
LeFusion-J (Ours)
P+N
69.87
37.31
59.56
36.19
Enhanced with Diffiasion Based Synthetic Mask (DiffMask)
Defusion-I+DiffMaee (Ours)
P+N'
69.81
40.62
58.94
39.00
LeFusion-J4DiffMaask (Ours)
P4N"
70.17
42.44
58.60
41.24
JeFusion-JDiffMassk (Ours)
PNP'+N"
70.34
43.54
60.54
41.70
LeFusion---HH+-iiMMask (Ours)
P+P'+N"
71.28
43.41
59.30
42.49
Cond-Diffusion Ho et al. 2020 Rombuch et al. 2022 These methods use the lesion mask and
background image information as conditional inputs (Rombach et al. (0022) diffusion model (Ho) al. 2020). However, a downside of this approach is that disrupts the background information. Furthermore, directly using multiple masks as conditional inputs fails to control the corresponding
categories, limiting the method modeling the generation of single-class lesions.
Cond- Cond-Diffusion (L) (Chen et al. 2024). Cond-Diffusion (L) conceptually a latent diffusion (Rom- bach et al.] (202)) version of Cond-Diffusia but adds VQGAN (Esser et al.] (2021) to map image into latent space for diffusion. For a fair comparison, we fine-tuned open-souree code and pre-trained weights by Chen et al. (024)) and used the model outputs directly.
LeFusion and the Variants (Ours). Apart from standard LeFusion, there are two variants for fine control of lesion textures. Histogram- Based Texture Control (*-H): A variant of LeFusion that incorporates histogram control information, using the input histogram to guide the generation of multi-peak lesion textures. Multi-Channel Decomposition (*-J): When handling multi-class lesions, standard LeFusion trains individual models separately (*-S for distinction), lacking of correlation modeling between classes. LeFusion-I is a generalized version to model multi-class lesions jointly.
4.2 IMPROVING SEGMENTATION WITH SYNTHETIC DATA
Lung Nodule Segmentation. We show that LeFusion can effectively benefit downstream ap- plication of training nnUNet (Isensee et al. 202T) and SwinUNETR (Hatamizadeh al. (02TT) to perform lung nodule segmentation. We use the following synthetic subset settings: P": 2104 synthetic ROIs from the 808 real pathological cases P: NO: 3076 synthetic samples from the 135 normal subjects N: N" 3076 synthetic cases from the 135 normal subjects N.
Tab. show the Dice and normalized surface distance (NSD). For the first group (Texture Synthe- sis with Real Masks), the texture of Hand-Crafted (Hu al. 2023) and RePaint (Lugmayr et al.) 2022 differs significantly from the real texture, making it challenging to achieve satisfactory results. Cond-Diffusion (Ho al.) 2020 Rombach al. 2022 and
(L) (Chen al.) 2024). on the other hand, disrupt the background structure of the generated images. Our baseline model, LeFusion, is impacted by the pixel distribution of the background due the lack of histogram
8

---

Published as a conference paper at ICLR 2025
Pathological
(4) Lung Nodule CT
Normal
Image
Mask
Hand- Cratt-
Cond- Oithaaon
Cand aond-omuton
RePaine
Pathological
(9) Cardiac Lasion MRI
Nomal
Coftee
Cand-Diilasin
Lafwian-S
Figure 5: Visualization of Synthetic Image on Emidec (2022) and LIDC (011). We compare the differences in image similarity between synthetic pathological cases generated by different methods, using real pathological cases and using normal regions. More visualizations in Appendix
trol information, resulting in only slight improvement over the baseline. We also synthesized lesion data on normal data N using Hand- Cund-Crafted Synthetic Masks and masks matched from lesion data, arriving at similar conclusions. In the final set of experiments, we validated the effectiveness of the DiffMask we designed for lesion synthesis and further explored accuracy improvements with increasing amounts of synthetic data. Compared to the baseline, in terms of Dice, we achieved im- provements 5.18% and 4.75% for nnUNet (Isensee et al. 2021) and SwinUNETR (Hatamizadeh et al. 2021), respectively. NSD improvements were 4.4% and 4.53%, respectively.
Cardiac Lesion Segmentation. We use the following synthetic subset settings P": 57 1 syn- thetic cases from the 57 real pathological cases P: N*: 33 x 2 synthetic cases from the 33 normal cases N: N 33 x 5 synthetic cases from the 33 normal cases N. We use combinations of these subsets to train nnUNet (Isensee et al. (021) and SwinUNETR (Hatamizadeh et al.) (021). The results of the two types lesions are reported in 2 in terms of the Dice (0-100, higher is better). Due to page limitations, the corresponding NSD table is provided in Appendix Tab, A1
In the first group of Tab. 2 we apply texture synthesis with real masks. The Hand-Crafter (Hu al. 2023) produces textures that differ from real textures, leading
decrease in baseline per- formance. Cond-Diffusion (Ho et al., 2020, Rombach et al. 2022) and
(L) (Chen) al. 2024) disrupts background structure, blurring lesion categories, decreasing MI's performance. RePaint (Lugmayr et al. 2022), focusing on global information, struggles to generate textures that conform to lesion characteristics, resulting in significant decrease in the Dice for PMO lesions. LeFusion models the two lesions separately, ignoring the correlation between lesions, which leads to improved accuracy for MI but decreased accuracy for PMO lesions. In contrast, our proposed LeFusion-J achieves superior results; For the second group, we expanded the normal data utilizing texture synthesis with hand-crafted synthetic masks. Due RePaint (Lugmayr et al.) 2022)'s ''s inabil- ity to distinguish between multiple lesion categories. we did not repeat experiments for it. From the experiments, we observed results similar to those of the first group; For the last, we evaluated our proposed lesion mask synthesis. Our method significantly improved the performance for both MI and PMO. Additionally, as data volume expanded, segmentation Dice consistently improved.
4.3 VISUAL QUALITY ASSESSMENT
Image Quality. Fig. 5 shows the generation results of different methods for lung nodule CT and cardiac MRI based on lesion images and normal images. We have also quantitatively calculated and compared the paired similarity between our generated images and real images; more details can be found in Appendix D Fig. a)a) displays the synthesized visualization of lung nodules (red). The Cond-Diffusion method (Ho al. 2020 Rombach et al. (022) and Cond-Diffusion (L) (Chen al. 2024) disrupt the background structure. The lesions generated by Band-Crafted (Hu et al.) 2023) and RePaint (Lugmayr et al., 2022) fail capture texture information, such as the grayscale varia- tions characteristic of the lesions. Our baseline LeFusion, without histogram control information, is easily influenced by background features, resulting in the generation of relatively shallow lesions.

---

Published as a conference paper at ICLR 2025
Control
Corbrol
Contma 3
NoCortrol
Pairvise PSNR, 51.14
Pairwise SSIM 99.39
Pairvise PSNR: 34.90 Pairvise SSIM: :
Figure 6: Illustration of Histograms Control Effectiveness. GT shows healthy lung tissue with the red area indicating the mask of the generated lesion. Control 1.2, and are images generated under different histogram controls. Lower numbers indicate higher diversity in the generated images.
In contrast, our LeFusion-H can better utilize histogram information to control the generation of lesions. Fig. (5)b displays the synthesized visualization of two lesions for the heart: MI (blue) and PMO (red). The generation results for the heart show similar conditions. LeFusion-J more accu- rately captures the textures of the two heart lesions, providing smoother transitions than LeFusion. Due to the small contrast between lesions, LeFusion-J-H yields similar results.
Unpaired perceptual metrics such as FID (Heusel et al. 2017 KID (Binkowski et al., 2018), and SWD (Karras et al. 2018) primarily assess semantic-level similarity. However, their correlation with visual quality, particularly for medical images, limited Jayasumana et al. (2024). Despite these shortcomings, we include these metrics reference for future work, with detailed results provided in Appendix D Nevertheless, we believe that evaluating generated data in medical contexts should focus more the effectiveness of downstream tasks. While assessing synthetic image quality is meaningful, quantitatively comparing different methods remains a challenge.
Histogram Control Analysis. We studied the effect of histogram control the lung nodule dataset. The visualization results are shown in Fig. Under different histogram controls, the atten- uation ("transparency") of the generated lesions changes from shallow deep. Without histogram control information, the generated lung nodules tend to match the pixel distribution of the normal lung background, resulting in overly light lesions. We selected 100 subsets and used LeFusion-H and LeFusion to generate each sample twice, calculating the similarity between pairs using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), lower means more diverse. As shown in the figure, with histogram control, there is greater diversity between samples. We aslo quantitatively analyzed histogram control on lesion areas, revealing shifts in distribution that align well with observations (see Appendix for details).
Mask Quality. We also visualize synthetic lesion masks, as shown in Fig. Al and Fig. A2 Com- pared the hand-crafted masks (Hu al. (2023), our diffusion-generannn masks are closer the real masks and exhibit a more diverse range of shape patterns. More details can be found in Appendix A
CONCLUSION
In conclusion, we introduce LeFusion, a resion-focused diffusion model that recalibrates learning objectives lesion areas only. preserves background by integrating -erfardddifused background contexts into the reverse diffusion process. Our methodology is extended to handle challenging multi-peak and multi-class lesions, and further enhanced by generative model for lesion masks, significantly diversifying our synthetic data. We demonstrate that synthetic data generated by method can effectively boost the performance state-of-the-art models.
Data-centric machine learning is increasingly crucial across scientific fields (Reichstein al. 2019 Rodriguez et al. 2024| Kimanius et al. 2024). This study focuses on generating pathological abnor- malities from normal anatomical structures, addressing data bias Mittermaier al. (2023). While key in medicine, this data synthesis approach is also applicable to other domains with abundant nor- mal data but scarce anomalies, such industrial, environmental, and material science anomalies.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
All experiments in this study were conducted using publicly available datasets. No clinical trials were performed, and the study does not involve human subjects, new dataset releases, or issues related to privacy or security. There are no potential conflicts of interest or sponsorships associ- ated with this work. The paper does not address topics such as discrimination, bias, fairness, legal compliance, or research integrity. However, it is important to acknowledge that our model could potentially be misused to generate fraudulent medical images, such as fabricating lesions in healthy individuals to commit insurance fraud.
REPRODUCIBILIIT STATEMENT
To enhance reproducibility, we provide the core code at: https: hnttps:/github.com com.M/t3..ccm// LeFusion We are committed to fully open-soureing the code along with the corresponding pre- processed data. Detailed descriptions of data preprocessing, model implementation, experimental settings for our approach and comparison methods (including batch size, learning rate, and check- point selection), as well as downstream evaluation procedures, are included Appendix We hope these measures facilitate reproducibility and encourage further research in this area.
ACKNOW.FDGEEMENT
This work was supported by Swiss National Science Foundation grant.
REFERENCES
Samuel G Armato III, Geoffrey McLennan, Lue Bidaut, Michael F McNitt-Gray, Charles Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. The lung image database consortium (lide) and image database resource initiative (idri): a completed reference database of lung nodules ct seans. Medical physics, 38(2):915-931, 201 1.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Benjamin Billot, Douglas N Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput, Bruce Fischl, Adrian V Dalca, Juan Eugenio Iglesias, et al. Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining. Medical Image Analysis, 86:102789, 2023.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations, 2018
Qi Chen, Xiaoxi Chen, Haorui Song, Zhiwei Xiong, Alan Yuille, Chen Wei, and Zongwei Zhou. To- wards generalizable tumor synthesis. In Conference on Computer Vision and Pattern Recognition, pp. 11147-11155 158, 2024.
Richard Chen, Judy Wang, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Ming Y Lu, Sharifa Sahai, and Faisal Mahmood. Algorithmic fairness in artificial intelligence for medicine and healthcare. Nature biomedical engineering, 76):719-7422 2023.
Yuhao Du, Yuncheng Jiang, Shuangyi Tan, Xusheng Wu, Qi Dou, Zhen Li, Guanbin Li, and Xiang Wan. Arsdm: colonoscopy images synthesis with adaptive refinement semantic diffusion models. In International conference medical image computing and computer-assisted intervention, pp- 339-349, Springer, 2023.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Conference on Computer Vision and Pattern Recognition, pp. 12873-12883, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11 13:9 2020.
11

---

Published as a conference paper at ICLR 2025
Ibrahim Ethem Hamamci, Sezgin Er, Anjany Sekuboyina, Enis Simsar, Alperen Tezcan, Ayse Gul- nihan Simsek, Sevval Nil Esirgun, Furkan Almas, Irem Dogan, Muhammed Furkan Dasdelen, et al. Generatect: Text-conditional generation of 3d chest ct volumes. In European Conference on Computer Vision, 2024.
Changhee Han, Yoshiro Kitamura, Akira Kudo, Akimichi Ichinose, Leonardo Rundo, Yujiro Fu- rukawa, Kazuki Umemoto, Yuanzhong Li, and Hideki Nakayama. Synthesizing diverse lung nodules wherever massively: 3d multi-conditional gan-based et image augmentation for object detection. In International Conference on 3D Vision, pp- 729-737. IEFE, 2019.
Colin Hansen, Simas Glinskis, Ashwin Raju, Micha Komreich, JinHyeong Park, Jayashri Pawar, Richard Herzog, Li Zhang, and Benjamin Odry, Inpainting pathology in lumbar spine mri with latent diffusion. arXiv preprint
2024.
Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pp. 272-284. Springer, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 13:6840-6851, 2020.
Izabela Horvath, Johannes Pactzold, Oliver Schoppe, Rami Al-Maskari, Ivan Ezhov, Suprosanna Shit, Hongwei Li, Ali Ertürk, and Bjoern Menze. Metgan: generative tumour inpainting and modality synthesis in light sheet microscopy. IEEE Winter Conference on Applications of Computer Vision, pp. 227-237, 2022
Qixin Hu, Yixiong Chen, Junfei Xiao, Shuwen Sun, Jieneng Chen, Alan Yuille, and Zongwei Zhou. Label-free liver tumor segmentation. In Conference on Computer Vision and Pattern Recognition, pp. 7422-7432, 2023.
Hussein Ibrahim, Xiaoxuan Liu, Nevine Zariffa, Andrew D Morris, and Alastair K Denniston. Health data poverty: an assailable barrier to equitable digital health care. The Lancet Digital Health, 3(4):e260-e3) e265, 2021.
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu- net: self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):203-211, 2021.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards better evaluation metric for image generation. In Conference on Computer Vision and Pattern Recognition, pp. 9307-9315 2024.
Qiangguo Jin, Hui Cui, Changming Sun, Zhaopeng Meng, and Ran Su. Free-form tumor synthesis in computed tomography images via richer generative adversarial network. Knowledge-Bate Systems, 218:10675: 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Firas Khader, Gustav Müller-Franzee Soroosh Tayebi Arasteh, Tianyu Han, Christoph Haarburger, Maximilian Schulze- Hagen, Philipp Schad, Sandy Engelhardt, Bettina Baeßler, Sebastian Foer- sch, et al. Denoising diffusion probabilistic models for 3d medical image generation. Scientific Reports, 13(1):7303, 2023.
Dari Kimanius, Kiarash Jamali, Max E Wilkinson, Sofia Lövestam, Vaithish Velazhahan, Takanori Nakane, and Sjors HW Scheres. Data-driven regularization lowers the size barrier of cryo-em structure determination. Nature Methods, pp. 1-6, 2024.
12

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv Preprint, 2013.
Yuxiang Lai, Xiaoxi Chen, Angtian Wang, Alan Yuille, and Zongwei Zhou. From pixel to cancer:
Cellular automata in computed tomography. arXiv Preprint, 2024.
Alain Lalande, Zhihao Chen, Thibaut Pommier, Thomas Decourselle, Abdul Qayyum, Michel Sa- lomon, Dominique Ginhac, Youssef Skandarani, Arnaud Boucher, Khawla Brahim, al. Deep learning methods for automatic evaluation of delayed enhancement-me the results of the emidec challenge. Medical Image Analysis, 79: 102428, 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Fei Lyu, Mang Ye, Jonathan Frederik Carlsen, Kenny Erleben, Sune Darkner, and Pong C Yuen. Pseudo-label guided image synthesis for semi-supervised 1ovid-19 pneumonia infection segmen- tation. IEEE Transactions Medical Imaging, 42(3):797-809, 2022a.
Fei Lyu, Mang Ye, Andy Ma, Terry Cheuk-Fung Yip. Grace Lai-Hung Wong, and Pong C Yuen. Learning from synthetic ct images via test-time training for liver tumor segmentation. IEEE Transactions on Medical Imaging, 41(9):2510-2520, 2022b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Mirja Mittermaier, Marium M Raza, and Joseph C Kvedar. Bias in ai-based models for medical applications: challenges and mitigation strategies. NPJ Digital Medicine, 6(1):113, 2023.
Wei Peng, Tomas Bosschieter, Jiahong Ouyang, Robert Paul, Edith V Sullivan, Adolf Pfefferbaum, Ehsan Adeli, Qingyu Zhao, and Kilian M Pohl. Motddita--onnditooeed generative models to syn- 3d brain mris. Medical Image Analysis, 98: 103325, 2024.
thesize matoomccally-plasstii
Alexander K Pishva, Vajira Thambawita, Jim Torresen, and Steven Hicks. Repolyp: framework for generating realistic colon polyps with corresponding segmentation masks using diffusion mod- els. In 2023 IEEE 36th International Symposium on Computer-Bool Based Medical Systems (CBMS), pp. 47-52. IEEE, 2023.
W Nicholson Price and Glenn Cohen. Privacy in the age of medical big data. Nature medicine, 25 (1):37-43, 2019.
Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha. Secure and robust machine learning for healthcare: A survey. IEEE Reviews in Biomedical Engineering, 14:156-180, 2020.
Markus Reichstein, Gustau Camps-Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Car- valhais, and F Prabhat. Deep learning and process understanding for data-driven earth system science. Nature, 166(7743),195-204, 2019.
Alexander Rodríguez, Harshavardhan Kamarthi, Pulak Agarwal, Javen Ho, Mira Patel, Suchet Sapre, and B Aditya Prakash. Machine learning for data-centric epidemic forecasting. Nature Machine Intelligence, pp. 1-10, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Pouria Rouzrokh, Bardia Khosravi, Shahriar Faghani, Mana Moassefi, Sanaz Vahdati, and Bradley J Erickson. Multitask brain tumor inpainting with diffusion models: A methodological report. arXiv preprint arXiv:2210. 12.22/0..22222 /2/13, 2022.
Neil Savage. Synthetic data could better than real data. Nature, 2023. URL fattps://dom
org/10 103//0. 01011/1000000/4111---033333-3330
13

---

Published as a conference paper at ICLR 2025
Raphael Schäfer, Till Nicke, Henning Höfener, Annkristin Lange, Dorit Merhof, Friedrich Feuer- hake, Volkmar Schulz, Johannes Lotz, and Fabian Kiessling. Overcoming data scarcity in biomed- ical imaging with foundational multi-task model. Nature Computational Science, pp. 1-15, 2024.
Younghak Shin, Hemin Ali Qadir, and llankko Balasingham. Abnormal colon polyp image synthesis using conditional adversarial networks for improved detection performance. IEEE Access, 6: 56007-56017, 2018.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2202222566662 PMLR, 2015.
Hualin Wang, Yuhong Zhou, Jiong Zhang, Jianqin Lei, Dongke Sun, Feng Xu, and Xiayu Xu. Anomaly segmentation in retinal images with poisson-blending data augmentation. Medical Im- age Analysis, 81:102534, 2022.
Linshan Wu, Jiaxin Zhuang, Xuefeng Ni, and Hao Chen. Freetumor: Advance tumor segmentation via large-scale tumor synthesis. arXiv Preprint, 2024.
Jie Xu, Yunyu Xiao, Wendy Hui Wang, Yue Ning, Elizabeth A Shenkman, Jiang Bian, and Fei Wang. Algorithmic fairness computational medicine. EBioMedicine, 84, 2022.
Jie Yang, Siqi Liu, Sasa Grbic, Arnaud Arindra Adiyoso Setio, Zhoubing Xu, Eli Gibson, Guillaume Chabin, Bogdan Georgescu, Andrew F Laine, and Dorin Comaniciu. Class-aware adversarial lung nodule synthesis in ct images. In International Symposium on Biomedical Imaging, pp- 1348-1352. IEEE, 2019.
Yuzhe Yang, Yuan Yuan, Guo Zhang, Hao Wang, Ying-Cong Chen, Yingcheng Liu, Christopher G Tarolli, Daniel Crepeau, Jan Bukartyk, Mithri R Junna, et al. Artificial intelligence-enable de- tection and assessment of parkinson's 's disease using nocturnal breathing signals. Nature medicine, 28(10):2277-2215, 2022.
Yuzhe Yang, Haoran Zhang, Judy W Gichoya, Dina Katabi, and Marzyeh Ghassemi. The limits of fair medical imaging ai in real-world generalization. Nature Medicine, pp. 1-11, 2024.
Qingsong Yao, Li Xiao. Peihang Liu, and S Kevin Zhou Label-free segmentation of covid-19 lesions lung IEEE Transactions on Medical Imaging. 40(10))2808-22) 2819, 2021.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 1505):10755- 2023.
Lingting Zhu, Noel Codella, Dongdong Chen, Zhenchao Jin, Lu Yuan, and Lequan Yu. Generative enhancement for 3d medical images. arXiv Preprint, 2024.
14

---

Published as a conference paper at ICLR 2025
A MASK VISUALIZATION
Fig. Al shows the real lung nodule lesion mask (a), the hand-crafted synthetic lung nodule mask (b), and the mask generated by our proposed diffusion model (d). Comparing subfigures (a), (b), and (d) in [ig.AA]] is evident that the real masks exhibit diverse shapes, while the DiffMask-generated masks closely resemble the real ones, also displaying varied forms with similar characteristies. In contrast, the handcrafted masks are relatively uniform in shape and differ significantly from the real
masks.
Besides, our proposed DiftMask can control the size and location of the lesion masks, allowing for the generation of more diverse lesions. As shown in Fig. [A] (c) and Fig. [Al (d), we can use the sphere (Size Ball) to precisely control the desired lesion size and its position within the background image. This control enables to produce various masks.
(a) Real
(c) Size Ball
(b) Hand-Crafted
(d) DiffMask
Figure A1: Visualization of Real / Synthetic Lesion Masks of Lung Nodule. (a) and (b) show the real masks and hand-crafted masks, (c) displays the control sphere (size ball) used by our proposed DiftMask, and (d) shows the corresponding synthetic mask results generated by DiftMask.
diffusion-generraeed Fig. A2 shows that our diffusion generated masks are closer to the real masks and exhibit a more diverse range of shape patterns, while Hand-Crafted masks consistently show large, continuous regions.
Real
Crad-Cr
DiffMask
Figure A2: Visualization of Real Synthetic Lesion Masks of Cardiac Lesion
B DOWNSTREAM SEGMENTATION PERFORMANCE
Tab. AT as a supplement to Tab. presents the NSD metric measured under the same experi- mental settings as in In Tab.
we employ texture synthesis with real masks. The Hand-Crafted ap- proach (Hu et al. 2023) generates textures that deviate from real ones, resulting in a decline
In the first set of Al
baseline performance. Cond-Diffusion (Ho al.) 2020) Rombach et al., (002) and Cond- Diffusion(L)Chhnn et al.) 2024) disrupt the background structure, leading to blurring between le- sion categories and reducing MI's performance. RePaint (Lugmayr et al. 2022), which emphasizes global information, struggles to produce textures consistent with lesion characteristics, leading to marked decrease in NSD for PMO lesions. LeFusion-S models the two lesions independently. disre- garding the correlation between them, resulting in higher accuracy for MI but reduced accuracy for PMO lesions. Conversely, our proposed LeFusion-J outperforms the other methods; In the second group, extended the normal data by synthesizing textures using hand-crafted synthetic masks.
15

---

Published as a conference paper at ICLR 2025
Table Al: Downstream Cardiac Lesion Segmentation NSD %) (1) on Emidee 2022). MI
and PMO are lesion classes. P: real pathological cases. P'/N:: synthetic pathological cases from pathological/normal cases. N more synthetic data than N'. Bold numbers indicate the best, with red highlighting significantly adverse effeets (> 1% lower) compared to the bascline, and blue
indicating significantly positive effects (> 1% higher).
Training Setting
nnU-Net (2021)
SwinUNETR (2021)
Methods
MI NSD (t) PMO NSD (T)
MI MINSD (1)
PMO NSD (1)
Bascline
P
59.27
29.19
47.66
20.51
Texture Synthewis with Real Masks p
Hand-Crafted (2023)
P+P
60.34
35.03
47.70
19.38
Cond-Diffusion 2020
2022
P+P"
58.21
24.04
46.35
20.87
Cond-Diffusion (L)
P+P'
58.98
25.96
46.83
18.72
RePaint (2022)
P+P
59.79
23.61
45.10
19.19
LeFusion-S S (Ours)
P+P'
60.77
33.13
46.66
20.05
LeFusion-J (Ours)
P+P'
60.44
36.65
48.23
24.11
Texture Synthesis with Hand-Crafted Syntheric Masks (2023) on N
Hand-Crafied (223)
P+N'
57.88
25.00
47.40
19.82
Rond-Diffusiin 2020
2022
P+N
58.22
22.19
46.15
20.30
Cond- Diff'usion L) (0244
P+P
58.64
26.26
47.90
19.54
LeFusion-S (Ours)
P4N'
60.75
23.96
47.81
19.91
LeFusion-J (Ours)
P+N
60.68
30.62
49.69
20.71
Enhanced with Diffission Based Synthetic Mask (DiffMask)
LeFusion-J+DiffMess (Ours)
P+N
61.35
38.93
48.62
22.43
LeFusion-J+DifMMask (Ours)
P+N"
61.48
35.03
50.00
23.92
Lefusion-JADillMaak (Ours)
P4P'+N"
61.27
41.62
52.82
23.77
LefuusionJI-H+-iifMMas (Ours)
P+P'+N"
62.74
40.96
50.73
24.25
Due to RePaint (Lugmayr et al. 2022)'s limitations in distinguishing multiple lesion categories, did not conduct repeated experiments for it. The outcomes were similar to those of the first group: Lastly, evaluated our proposed lesion mask synthesis. Our approach significantly enhanced the performance for both MI and PMO. Furthermore, as the data volume increased, the downstream segmentation NSD showed consistent improvement.
C HISTOGRAM CONTROL ANALYSIS
For given original image, the histogram information of its lesion region is represented as 1j. and the control information as h1, h2, h3..... .., hn (where i = 1..., n), with n being the number of controlled histograms. The resulting output image is defined as O, and O can be derived as follows:
......,,,
m, (r log +p) + x log
0= mh i= 1,2.3......
In this formula, and are scaling factors, and and are bias offset terms. Based on the above formulas, given the input image and corresponding control information, we can theoretically deduce the histogram of the lesion in the mask area of the generated image.
Fig. A3
shows the impact of histogram control information on the generation lung nodules normal chest background. We randomly sampled 20 cases from the generated lesion data for statis- tical analysis. The first row represents the controlled histogram, where "No control" indicates that the standard diffusion configuration without histogram control information row shows the original mask areas of the 20 cases, as well as the average histogram effects of the generated and predicted data. Rows 3, and 5 display the results of three different sample cases.
used. The second
16

---

Published as a conference paper at ICLR 2025
Control 1
Control
Control 3
Contrll
Avg.
Case 1
Case 2
Case 3
(a) Texture GT
(b) Result Histograms
Figure A3: Illustration of Histograms Control Effectiveness. The light blue histogram on the far left represents the original mask area. The pink histograms in the first row indicate the control formation applied by the diffusion model. The yellow histogram represents the theoretically derived output lesion area, while the darker blue shows the actual generated lesion histogram.
When the model does use histogram control, the generated histograms of lung nodules in the normal background areas tend to be relatively shallow, influenced by the surrounding background. These histograms resemble the textures of the background mask areas, as shown in the rightmost column. In this column, the dark blue represents the histogram of the image mask area generated by the diffusion model, while the orange represents the theoretically fitted histogram control effect.
After introducing histogram control information, the interaction between the original image lesion areas and the controlled histogram effects causes the generated mask areas to shift increasingly to the right, following the peak effects of the three histograms (control 1, control 2, control 3). The second-row average histogram effect shows the stable trend of this shift.
D IMAGE QUALITY EVALUATION
As shown in Tab. A2 we selected Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) to calculate the similarity between synthetic pathological cases generated by different methods and real pathological cases. for cardiac lesions on the EMidec dataset, our proposed LeFusion achieved the highest average PSNR and SSIM. For lymph nodes with only one type of lesion on the LIDC dataset, LeFusion also achieved the highest PSNR. We have also included the Fréchet Inception Distance (FID) (Heusel al. 2017) and Kernel Inception Distance (KID) Binkowski et al. 2018 metrics, as shown in Tab. A3 and the Sliced Wasserstein Distance (SWD) Karras al. 2018) in Tab. A4 These results align with the similar phenomena discussed above. These experiments demonstrate that our synthesized lesions are more similar to real lesions across both CT and MRI modalities.
17

---

Published as a conference paper at ICLR 2025
Table A2: Paired Synthesized Image Quality Assessment PSNR (1) and (S)M(%) (f) on Emidec (2022) and LIDC (201T). We compare the differences in image similarity between syn- thetic pathological cases generated by different methods given real pathological cases.
Emidec-MI
Emidec-PMi
Emidec-Avg.
LIDC
Methods
PSNR
SSIM
PSNR
PSNR
SSIM t
PSNK 1
SSIM
Hand-Crafied (2023)
9.39
8.30
7.63
8,516
8.70
0.07
Cond-Diffusion 2020 2022
13.25
46.92
8.00
9.23
10.62
28.07
16.95
93.46
L) 2024
14.62
69.36
12.39
61.00
13.51
65.18
15.50
90.05
RePaint 2022
19.81
80.68
15.23
70.27
17.52
75.47
18.91
91.22
LeFusion-S (Ours)
25.65
91.78
27.71
26.68
90.60
22.38
90.16
LcFusion-J (Ours)
28.30
91.41
35.23
93.23
31.77
92.32
Table A3: Unpaired Synthesized Image Quality Assessment of FID (%) ()) and KID (%) (J) Emidec (2022) and LIDC (2011). We compare the differences in image similarity between synthetic pathological cases generated by different methods given real pathological cases.
Emidec-MI
Emidec-PMM
Emidec-Ave
LIDC
Methods
FID 4
KID 4
FID 4
KID 4
FID 4
KID 4
FID 4
KID !
Hand-Crafted 2023
19.06
3.58
17.67
16.75
18.36
10.17
12.22
2.57
2020 2022
12.14
1.79
17.18
11.43
14.66
6.61
6.99
0.86
Cond-Diffusion(:
2004
12.38
1.94
22.92
9.71
17.65
5.83
9.13
1.54
RePaint 2022
17.69
3.94
15.49
15.67
16.59
9.80
9.33
0.84
LeFusion-S S (Ours)
7.09
1.31
5,21
4.01
6,15
2.66
6.42
0.73
LeFusion-J (Ours)
5.39
0.78
4.15
0.50
4.77
0.64
E MORE VISUALIZATIONS
In this section, we provide multiple illustrative figures demonstrating the effects of our proposed diffusion model, LeFusion.
Different Recurrent Length Effects. On the one hand, the recurrent operation enhances the sta- bility texture generation. For instance, employing a medium loop achieves more effective his- togram control compared to no recurrent operation, particularly when histogram values rela- tively low, as demonstrated in the first and second rows. On the other hand, the recurrent approach improves smoothness and semantic coherence in the intersecting regions between the lesion and the background. For example, in the first row and the first column, the larger loop more effectively cap- the spiculated features of the lung nodule, whereas the no recurrent approach tends to overlook the correlation between the lesion and the background.
Table A4: Unpaired Synthesized Image Quality Assessment SWD (1e-4) (!) on Emidec (2022) and LIDC (2011). We compare the differences in image similarity between syn- thetic pathological cases generated by different methods given real pathological cases.
Methods
Emidec-MI
Emidec-PMO
Emidec-Avg,
LIDC
Cand-Crafted (2023)
26.62
4.13
15.38
10.64
Cond-Diffusion (2020 2022
26.51
5.24
15.88
6.64
Cond-Diffusion (L) (2024
15.83
5.04
10.43
7.95
RePaint (2022)
13.75
2.93
8.34
11.64
LeFusion-S (Ours)
11.62
2.97
7.29
5.90
LeFusion-J (Ours)
9.94
1.60
5.77
18

---

Published as conference paper at ICLR 2025
No Recurrent
Small Loop
M Medium Loop
Large Loop
No Recurrent
Small Loop Medium Loop
Large Loop
Control
Control
Control
Figure A4: Visualization of Different Recurrent Length Effects. "No Recurrent" indicates no recurrence during lesion generation. "Small," "Medium," and "Large" represent varying levels of time step skipping and cyclic folding, ranging from short to long durations. "Control11" "Control2," and "Control3" refer to the generation effects under three different histogram controls.
Lung Nodule CT. Fig. A5 presents additional illustrations of the effects of histogram control. The histogram effectively controls the texture of the lesions, while the version without control in- formation tends to generate lighter-colored lung nodules. Fig. A6 provides a visualization of the synthesized lesion results using real samples and their corresponding normal samples.
GT
Control 1
Cantrol 2
Control 3
Nontorool
Figure A5: Illustration Histograms Control Effectiveness. GT shows healthy lung tissue with the red area indicating the mask of the generated lesion. Controll, Control2, and Control3 are images generated under three different histogram controls.
Cardiae Lesion MRI. Fig. A7 shows the visualization of the denoising process at different stages LeFusion for inpainting. Fig. A8 and Fig. A9 respectively show the generation of pathological results on lesion cases and normal cases.
19

---

Published as a conference paper at ICLR 2025
Patholog loal
Normal
Normal Image
Mask
Hand-Crafted
Cond- Diffusion
Cond- Doff-Diffuion (L)
RePaint
LeFusion
LeFusion-H
Figure A6: Visualization of Synthetic Image LIDC (Armato III et al., 2011 We compare the differences in image similarity between synthetic cases generated by different methods, using real pathological cases and normal regions.
Image
Mask
Input
Dencising 0%
Denoising 60%
Denoising 75%
Figure A7: Visualization the Denoising Process in LeFusion for Inpainting. The process is conditioned the masked input. Starting from random Gaussian noise sample, the procedure iteratively denoises this input, progressively refining into a high-quality image with the lesions.
F IMPLEMENTATION DETAILS
For the entire experiment, we used 6 A*A10 (40G) GPUs, including for diffuusion and downstream segmentation tasks, with Python 3.8 and PyTorch version 2.4.0.
Datasets. For lung nodules, we followed the common practice uniformly rescaling the spacing 1.0 x 1.0 x 1.0 lesion to a fixed size of 64 x 64 x 32. For cardiac lesions, since the overall data quality was not high minimal, we retained the original spacing Furthermore, we uniformly cropped and padded each lesion to fixed size of 72 x 72 x 10.
(Han
2019). Additionally, we uniformly cropped and padded each
and the variation in spacing
data precision.
Diffusion Models. Training the LeFusion diffusion model for lung nodules requires approxi- mately A100 GPUs for one day. For inference, generating a single data takes about 30-50 seconds on one A100 GPU.
Recurrent Mechanism for Inpainting. To achieve better consistency between lesions and back- ground, aim to enhance the integration of lesion and background information by allowing the generated images to undergo recurrent processes within the model. Specifically. we adopted some commonly used techniques from the inpainting field (Meng al. 2022) (Lugmayr al. 2022 We
20

---

Published as a conference paper at ICLR 2025
Image
Mask
CradtCr
Lind-Diflusion
Repaint
LeFusion-S
LeFusion-]
Figure A8: Visualization of Synthetic Images Given Real Lesion Masks on Pathological Cases. Our LeFusion are compared with Sranf-C Hu et al. (2023), Cond-Diffusion Ho et al. (2020); Rombach et al. (2022) and RePaint I Lugmayr et al. (2022).
Image
Mask
Hand-Crafted
Cond- Diffusion
Repaint
LeFusion-S
LeFusion-J
Figure A9: Visualization Synthetic Images Given Synthetic Lesion Masks on Normal Cases.
Our LeFusion compared with Hand-Crafted Hu et al. 2023). Cond-Diffusion Ho et al. 2020
Rombach et al. (202) and RePaint Lugmayr et al. (2022).
defined two key parameters: the recurrent length and the recurrent sampling frequency. The recur- rent length refers to the time span of the recurrent operation, where a larger recurrent length can integrate more information from both the lesion and the background. The recurrent sampling fre- quency refers to the number of times each recurrent point is sampled. For example, if the initial time step is 300, the recurrent length is set to 2 (skipping two time steps), and the recurrent sampling fre- quency (each jump point revisited once), then the sequence of time steps during the diffusion process would be as follows:
(300, 299, 298, 299, 300, 299, 298, 297, 296, 298, 297, 296, 295, 294, 295, 296, 295, 294, 293...)
Selection of Histograms. When generating lesions, the control information for each lesion is randomly selected from three control levels (control3, control2, controll) the ratio 75:20:5. We also experimented with other ratios, such 1:1:1 3:2:1, and found that the performance on downstream tasks was roughly the same. To enhance the diversity of the generated lesions, we introduced fluctuation mechanism for each component of the control information. This mechanism allows the value of each component to randomly vary within + +10% of its original value, while ensuring that the sum of the components remains equal 1.
21

---

Published as a conference paper at ICLR 2025
Comparison Method Details. For the diffusion model architectures compared in pa- per RePaint, Cond-Diffusion, and Cond-Diffusion (L)-al all share a similar U-shaped structure. The primary difference between Cond-Diffusion and RePaint lies in their channel configurations, with Cond-Diffusion (L.) incorporating latent features as input. In our experiments, we observed that the convergence speed is nearly identical across these models. Therefore, ensure experi- mental fairness, we used a unified configuration for all diffusion models. Specifically, all diffusion 300 timesteps. For both datasets, adopted a learning rate of le-4 and batch size of 16. To ensure that each diffusion model fully converged, we chose as many training epochs as necessary ensure the training loss remained stable without continuing decrease. The training process required approximately 30,000 timesteps for the cardiac dataset and 40,000 timesteps for the LIDC lung nodule dataset.
models were
Segmentation Models. We implemented nnUNet [sensee et al. 2021 and Swin- UNETR (Hatamizadeh et al. (021) using the MONAI framework. For downstream tasks, both SwinUNETR and nnetet were trained for 200 epochs. Due differences in dataset sizes, the training time single A 100 GPU was approximately 6 to 24 hours for SwinUNETR and 4 to 10 hours for unet.
22


Reviews:
[{"review_id": "kr7KYzUpIE", "reviewer": "Reviewer_afPU", "summary": "This manuscript presents a diffusion model that utilizes forward-diffused backgrounds and reverse-diffused foregrounds as inputs, allowing the model to concentrate on reconstructing lesions specifically. Additionally, a post-processing method is applied to enhance generation quality.", "questions": "Please revise Figures 1 and 2 to more clearly illustrate the novelty of your proposed approach. Rather than emphasizing the strengths of the paper or incorporating numerous elements into a single pipeline, focus on presenting a straightforward and cohesive pipeline that highlights the mechanisms unique to your method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This manuscript is well-motivated, and the experimental results are satisfactory.", "weaknesses": "There are several concerns regarding this manuscript:\n\n* The novelty of the proposed approach is limited. The method does not significantly modify the underlying conditional diffusion process but instead introduces variations solely in the input.\n* Figure 2 lacks clarity, and it would be beneficial to include the lesion-focused loss in this figure for a more comprehensive understanding.\n* The writing lacks organization and is difficult to follow, which may impede readability and comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MLc9jxKbT3", "reviewer": "Reviewer_rF6W", "summary": "The authors introduce a latent diffusion model-based method for inserting lesions into healthy medical images while also providing an accompanying mask. They utilize a number of additions to their model to address limitations of prior work or naïve approaches to this task (both pre-existing and seemingly novel), such as combining forward-diffused backgrounds with reverse-diffused foregrounds, introducing intensity histogram-conditioning to the diffusion model to control lesion texture, as well as techniques for further control of the shape, size etc. of the generated lesion. They evaluate their method for a variety of experimental scenarios on 3D cardiac MRI lesion and CT lung nodule generation, showing that their technique results in noticeable improvements to existing approaches with respect to using their generated data to train downstream task segmentation models.", "questions": "1. In the tables (e.g. table 1), what do you mean by the significantly adverse/positive effects denoted by red/blue? Could you please clarify this in the text as well via a small note in the table caption(s)?\n2. My suggestion: move image quality assessment quantitative results in the appendix (Table A2) to the main text if you have room. These are important metrics. You can shorten the related works to make space, that section doesn't need to be quite so extensive (or some of it could be moved to the supplementary).\n    - Also, why didn't you evaluate unpaired perceptual metrics like FID, KID (https://arxiv.org/abs/1801.01401), SWD (https://arxiv.org/abs/1710.10196) etc.? the first two may have limitations for this task given that they use pretrained natural image features, but despite this they are still commonly used metrics for generative medical image models. I would consider adding these for future work, and also explaining why they are not used (particularly for the wider ICLR audience).\n3. For the multiclass lesion case/-J model, did you study how performance/generation quality scales with adding more classes? This point may be a bit moot given how small the changes in performance were measured after adding the channel decomposition module to the base model, but I'm still curious.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Major\n1. The paper is polished, well-written and well-presented. Topics and concepts are organized and presented in a digestible fashion.\n2. Overall, decent technical novelty. This incorporates many techniques which all come together to result in a strongly-performing methods, some pre-existing (such as combined noised backgrounds with denoised foregrounds), and some seemingly novel (such as histogram-based textural control). Also, despite the many components, the approach still seems relatively watertight because these additions are all pretty lightweight/simple (a good thing). No requirement for an additional network or something of that sort.\n3. Overall, results are strong. Clear improvements over baseline methods is basically all cases, using reasonable metrics. They also study a range of training settings, which is good. Clear improvements over Cond-Diffusion, which would be the naïve approach that many would think of first trying for this task; the limitations of it as discussed in the introduction are clear from the experiments.\n4. They also have fairly extensive ablation studies for their method, which is important given the number of components that they propose using. There are still a few related questions that I have, but they are minor.\n5. In general, the evaluation is fair and appropriate. The datasets are challenging benchmarks, and I think two is sufficient given the wide range of experiments completed on them. There is also a good number of baseline models, especially considering that this task is relatively niche, so the methodological baselines that they compare to seem strong.\n\nMinor\n1. The motivation for this problem is clear: pathological subjects are indeed rare, especially for screening populations. Your survey of the limitations of existing lesion synthesis approaches also supports the motivation; for example, they result in low quality backgrounds, they lack precise control over generated lesions, etc.\n2. The use of a histogram representation to condition the model on may seem too reductive for some applications, but it seems to work well here (makes sense given the clear correspondence between histogram shape/number of peaks and generated lesion morphology shown in Fig. 3), supported by the clear improvement to your method that including the -H module produced.", "weaknesses": "Major\n1. Some limitations of impact/scope: This task is clinically important but still fairly niche in medical image analysis, which itself is fairly niche within general machine learning and computer vision. The method (and task itself) also requires that dataset used needs the required annotations, which many medical datasets may not possess, and can be expensive/time-consuming to acquire. Overall, these limit the impact of the work somewhat, in the context of an ML conference at the level of ICLR, compared to a venue a bit more niche like MICCAI.\n\nMinor\n1. The benefits from using multi-channel decomposition (comparing the \"-J\" to no \"-J\" variants of your model in Table 2) are quite small. Can you provide some analysis or discussion of why this is the case, even if just hypothesizing? (However, I am guessing that the computational requirement to adding this component is practically negligible, so there is not really any harm in including it even if it results in only a very small performance improvement.)\n2. You state in the abstract that synthesizing multi-peak and multi-class lesions is a \"major challenge\" I agree with the multi-peak case given how much your histogram-conditioning improved the generation of such lesions, but based on your channel decomposition module's only very small improvements to performance, I'm unsure if generating multi-class lesions could not already be done well by prior methods. Could you clarify this/point to your results that support this, and/or provide quantitative evidence that multi-class synthesis is challenging for prior approaches?\n\nTo summarize, the paper is methodologically solid, with some technical novelty, and demonstrates clear improvements to prior techniques for lesion generation tasks in medical images via well-designed experiments and baselines. However, the main limitation is just that the task is relatively niche within medical image ML, which makes it more niche within general ML, and so may be less impactful at a venue like ICLR as opposed to a medical imaging-focused venue such as MICCAI or MIDL. Still, these limitations do not take away the good things about the paper (of which there are many), so I vote for a marginal accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTV85eMNOt", "reviewer": "Reviewer_XM6f", "summary": "This paper introduces a novel 3D lesion inpainting method, LeFusion, which uses diffusion models to address data scarcity in medical imaging. Its primary aim is to generate synthetic lesions in lung CT and cardiac MRI scans for augmenting training data in lesion segmentation tasks. The approach is validated through both visual quality assessments and data augmentation derived segmentation \n\nperformance improvement. Three key contributions can be summarised below: \nLeFusion Model: The authors identify that existing lesion inpainting methods struggle to preserve anatomically accurate backgrounds alongside the inpainted lesion, remarking that modelling the former is both hard and unnecessary. LeFusion is introduced to address this challenge incorporating two distinct features: (a) Training on a lesion focused diffusion loss, which only considers the lesion region. (b) Preserving the background at inference time with RePaint [1] by generating the lesion separately, while integrating forward-diffused background contexts into the reverse diffusion process. This design yields realistic lesions, better preserved backgrounds and improves data augmentation outcomes in both CT and MRI compared to non-lesion-specific models (Cond-Diffusion) both with and without RePaint based sampling. \n\nModality-Specific Variants: Two specialized variants are introduced to address modality-specific challenges. LeFusion-H uses histogram-based conditioning to capture diverse lesion textures in CT, succesfully solving the texture mode collapse observed for the baseline LeFusion. LeFusion-J models multiple tissue subtypes in MRI via multi-channel decomposition, which enables the joint generation of different lesion tissue types typically observed in cardiac lesions. Both variants demonstrate superior data augmentation effectiveness in their respective modalities. \n\nDiffMask for Mask Generation: All variants of LeFusion rely on either existing real masks or handcrafted ones as priors for generating lesions in healthy scans. As a more flexible alternative, DiffMask is a diffusion model that generates synthetic lesion masks from basic spatial constraints, defined as a sphere with user specified location and size. Using the generated masks for data augmentation leads to the largest improvement in segmentation performance relative to the baseline in both CT and MRI.", "questions": "For specific questions please refer to the points made in weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Lesion generating models are tools with significant potential for mitigating bias in medical vision AI algorithms concerning lesion detection, segmentation and quantification. Advancements in this topic should be highlighted in venues like this. \nThe manuscript is sufficiently well written, all the provided Figures/Tables are insightful and adequately formatted. \nThe choice of a 3D method for this inpainting problem is most adequate for CT and MRI. In these modalities, clinical lesion analysis workflows depend on the visualisation of multiple affected slices and 2D slice-wise inpainting methods would lead to slice-wise discontinuities. \n\nThe proposed method is sufficiently contextualised in the Introduction and Related work sections, where the reseach gap is clearly defined. Beyond that, this gap is empirically demonstrated by experimenting with state-of-the-art approaches (Cond-Diffusion variants and RePaint). \n\nThe proposed methodologies are thoroughly evaluated through comparisons with multiple other approaches, focusing on visual inspection of inpainted lesions (including comparison with real lesions) and their their downstream usability for training segmentation models. The latter evaluation used two different segmentation models, which contributes to the robustness of the findings across different segmentation training strategies. In addition, evaluating the approach on both MRI and CT datasets, ensures that the findings are not only applicable to one imaging domain. \n\nThis paper provides multiple key contributions which not only address the research gap but also deal with modality specific challenges related to lesion texture and shape heterogeneity. The corresponding claims are well supported by the results.", "weaknesses": "While S4, the Introduction and Background sections seem to imply that the proposed lesion focused loss is a novel contribution proposed for the first time by the authors. This might not be necessarily true considering that there have been other works that employ similar approaches [2, 3]. While few and perhaps not as thoroughly evaluated, mentioning them could further strengthen the contextualisation of the approach. \n\nThe description of the RePaint method in the experimental section implicitly suggests it consists of Cond-Diffusion using the RePaint [1] inference scheme. If that is the case it should be mentioned explicitly, if not then it should be better described. \nIn the segmentation experiments, it is understood that masks priors for generating lesions in healthy scans (N’) are either derived from real masks, handcrafted or generated by DiffMask. However, additional information should be provided on how exactly the conditioning histograms in this N’ setting are selected when using LeFusion-H variants. \n\nRegarding DiffMask, the definition and role of boundary mask is not very clear. From Figure 4, it is presumed that it corresponds to the bounding box defining the volume crop centred on the lesion. However, the statement “The boundary mask removes areas outside the boundary at each diffusion step” challenges this concept. Further clarity on this point would be appreciated. Furthermore, it is only implicit, that the DiffMask takes the CT/MRI volume crop as an input in addition to the conditioning control sphere. Section 3.3. should be updated to enhance clarity on all these aspects. \n\nAdding supplementary details on how the model training and checkpoint selection was conducted for the RePaint, Cond-Diffusion, Cond-Diffusion (L) would improve transparency. \n \n[Minor]\t \nMore detail on the dataset preprocessing would be beneficial for further reproducibility. A mention to the volume resolution is particularly lacking. \n\nThe choice of the specific crop-size could be further supported on previous work, for instance [4]. In addition, while not critical for acceptance, it would be interesting to study its effect over the results and would maybe answer the question: “How much local context is it necessary to generate realistic lesion?” \n\nWhile the purpose of the inpainted lesions is for downstream model training, further validating them using a radiologist would safeguard from potential biases that the generative model might be introducing the lesions. \n\nWhile describing Tables 1 and 2 it would be useful to clarify what is considered as “significant”. Since no standard deviations were provided, it is implied that these results were obtained for a single fold, so the concept of significance here is vague. In addition, while S5, the robustness of these findings to the specific data split could still be reinforced by adopting some sort of cross validation strategy. \nThe authors left unclear whether the segmentation model was trained on the volume crops centred on the lesion or on the entire scans. From using the Copy-Paste method in the evaluation, the latter is presumed but it is not explicitly mentioned. \n\nIn the cardiac MRI experiments, the LeFusion baseline of modelling the two lesion tissue types with separate models is mentioned as LeFusion in Table 2 but as LeFusion-S in Figure 5 and in the Appendix. It is suggested that the authors stick to one terminology. \nAs a work mainly focusing on specific diffusion model mechanics for improved lesion inpainting, it makes sense that the evaluation focus on comparing different diffusion based methods. That said, it would still be interesting to see how GAN based approaches like [4, 5] would fair in this comparison. \n \nReferences:  \n[1] Lugmayr, Andreas, et al. \"Repaint: Inpainting using denoising diffusion probabilistic models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. \n[2] Hansen, Colin, et al. \"Inpainting Pathology in Lumbar Spine MRI with Latent Diffusion.\" arXiv preprint arXiv:2406.02477 (2024). \n[3] Rouzrokh, Pouria, et al. \"Multitask brain tumor inpainting with diffusion models: A methodological report.\" arXiv preprint arXiv:2210.12113 (2022). \n[4] Yang, Jie, et al. \"Class-aware adversarial lung nodule synthesis in CT images.\" 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019. \n[5] Wu, Linshan, et al. \"FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis.\" arXiv preprint arXiv:2406.01264 (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hmDmYcluKJ", "reviewer": "Reviewer_f4py", "summary": "This paper focuses on generating lesion-containing images from healthy images to address challenges in downstream segmentation tasks, such as real-world data scarcity and long-tail distribution issues. Previous research on medical image synthesis has primarily concentrated on lesion generation design, often overlooking high-fidelity background preservation. The authors propose a lesion-focused diffusion model, LeFusion, which maintains high-fidelity background by integrating the background from forward diffusion into the reverse diffusion process, thus simplifying the learning process and improving output control. Additionally, two effective strategies are introduced: histogram-based texture control and multi-channel decomposition to address the two main challenges in lesion texture synthesis: 1) multimodal and 2) multiclass lesions. The paper is well-written, with comprehensive experimental comparisons.", "questions": "1. The reverse diffusion sampling process is not clearly defined; it appears to rely solely on the transformation in Equation (1), without detailing the sampling process or providing theoretical justification for omitting it.\n2. Although the background from forward diffusion is used as the background in the reverse sampling process, and the loss constraint is applied only to the lesion area, how is continuity and smoothness ensured in the intersecting regions between the lesion and background?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall paper structure is clear and well-expressed.\n2. A novel diffusion model is redesigned from the perspective of high-fidelity background preservation, with two texture generation control techniques developed to address multimodal and multiclass issues.\n3. The comparative methods are recent benchmarks from the past two years, making the results highly convincing.", "weaknesses": "1.There is a lack of detail on implementation specifics (such as the sampling process) and theoretical support for the method.\n2. Analysis and discussion on the continuity at the fusion boundaries between lesion and background are missing, as well as the impact on downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)