LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models

## Abstract

Abstract Patient data from real-world clinical practice often suffers from data scarcity and long-tail imbalances, leading to biased outcomes or algorithmic unfairness. This study addresses these challenges by generating lesion-containing image-segmentation pairs from lesion-free images. Previous efforts in medical imaging synthesis have struggled with separating lesion information from background, resulting in low-quality backgrounds and limited control over the synthetic output. Inspired by diffusion-based image inpainting, we propose LeFusion, a lesion-focused diffusion model. By redesigning the diffusion learning objectives to focus on lesion areas, we simplify the learning process and improve control over the output while preserving high-fidelity backgrounds by integrating forward-diffused background contexts into the reverse diffusion process.
Additionally, we tackle two major challenges in lesion texture synthesis: 1) multi-peak and 2) multi-class lesions. We introduce two effective strategies: histogram-based texture control and multi-channel decomposition, enabling the controlled generation of high-quality lesions in difficult scenarios. Furthermore, we incorporate lesion mask diffusion, allowing control over lesion size, location, and boundary, thus increasing lesion diversity.
Validated on 3D cardiac lesion MRI and lung nodule CT datasets, LeFusion-generated data significantly improves the performance of state-of-the-art segmentation models, including nnUNet and SwinUNETR.
Code and model are available at https://github.com/M3DV/LeFusion .

## 1 Introduction

The development of AI for healthcare often suffers from data scarcity (Ibrahim et al., ; Schäfer et al., ). In most biomedical scenarios, the number of pathological subjects is significantly lower than that of normal ones. This discrepancy primarily arises from the naturally occurring distribution of patient data, which frequently exhibits long-tail characteristics (Yang et al., ; Zhang et al., ). Additionally, potential biases in data collection can introduce issues related to algorithmic fairness (Xu et al., ; Chen et al., ; Yang et al., ), as well as concerns about security and privacy (Price & Cohen, ; Qayyum et al., ). As a result, it has been argued that “synthetic data can be better than real data” (Savage, ).

Generative lesion synthesis is a promising approach to generating diverse medical data, benefiting many medical applications (Khader et al., ). By learning from lesion-containing data, generative models can synthesize various types of lesions, which in turn benefit downstream applications (Han et al., ; Jin et al., ; Yang et al., ; Lyu et al., ; Shin et al., ; Pishva et al., ; Du et al., ; Lyu et al., ). While a range of generative methods have been explored, they often struggle to preserve high-quality backgrounds outside of the lesion areas. This is because generating anatomically correct backgrounds in the human body is far more challenging than synthesizing isolated lesions. Moreover, these methods often lack control over key aspects of lesion generation, including texture type, size, location, and mask alignment. These issues can severely degrade the performance of downstream applications, such as segmentation algorithms trained using such synthetic data. Fig. 1a illustrates standard diffusion models as an example.

One approach to avoiding the complexity of background generation is to start from readily available normal scans and synthesize lesions into them. This involves generating lesion masks and filling them with appropriate textures, ensuring perfect background preservation and mask alignment, while also allowing precise control over lesion size and location. In this paper, we refer to this method as background-preserving lesion synthesis. This approach has led to a resurgence of hand-crafted methods, which have been used to model COVID-19 lesions (Yao et al., ) and liver tumors (Hu et al., ). However, these methods rely on heuristics that do not generalize well.

Inspired by diffusion-based image inpainting schemes (Lugmayr et al., ; Avrahami et al., ), it has been shown that explicitly integrating real background context during the diffusion process ensures realistic background preservation outside lesion masks. Rather than using the background as conditional inputs (Rombach et al., ), these methods directly incorporate forward-diffused background contexts into the reverse diffusion process. However, these approaches are training-free and do not focus on specific inpainted content. In contrast, our study emphasizes lesion generation. In data-limited scenarios, it is more efficient for the model to focus solely on lesion synthesis.

Figure: Figure 1: Standard Conditional Diffusion vs. Lesion-Focused Diffusion (LeFusion). (a) Standard Conditional Diffusion concatenates background, lesion mask, and noise. The model generates both lesion and background, risking background integrity and wasting capacity on difficult but unnecessary background generation, especially in data-limited settings. (b) Lesion-Focused Diffusion (LeFusion) uses forward-diffused backgrounds and reverse-diffused foregrounds as input. The model reconstructs only the lesion, ensuring realistic background preservation and simplifying the task. (c) LeFusion with Fine Control of Lesion Textures and Masks introduces histogram-based texture control for multi-peak lesions, multi-channel decomposition for multi-class lesions, and lesion mask diffusion for control over size, location and boundary, enhancing lesion quality and diversity.
Refer to caption: x1.png

To this end, we propose LeFusion, a lesion-focused diffusion model (Fig. 1b and Fig. 1c). We redesign the diffusion learning objectives to focus solely on lesion data. Similar to diffusion-based inpainting, the input combines forward-diffused backgrounds with reverse-diffused foregrounds, while the model reconstructs only the lesion, avoiding the need to allocate capacity to learning complex backgrounds.
Furthermore, we address two major unresolved challenges in lesion synthesis: 1) multi-peak lesions, where lesions have distinct types, and 2) multi-class lesions, where multiple classes of lesions need to be generated simultaneously. For the multi-peak challenge, we introduce histogram-based texture control, integrating lesion texture histograms during training as a condition, which allows control over lesion types during inference. We find that explicitly controlling the histogram is crucial when generating lesions on normal scans; otherwise, the model tends to produce lesions biased toward healthy appearances. Notably, this histogram-based control method is generic and does not require any additional information beyond image-mask pairs.
To handle the multi-class challenge, we propose a strategy for joint modeling of multi-class lesions through multi-channel decomposition, where the diffusion model generates different lesion classes via separate channels and then combines them into a single image.
Finally, we introduce lesion mask diffusion, enabling control over size, location and boundary, thereby increasing the diversity of lesion masks.

We validate LeFusion on 3D lung nodule CT datasets (Armato III et al., ) and cardiac lesion MRI (Lalande et al., ), demonstrating its effectiveness in addressing both the multi-peak and multi-class challenges while generating high-quality synthetic lesions. In downstream segmentation tasks, we show that LeFusion-generated data significantly enhances the performance of state-of-the-art models such as nnUNet (Isensee et al., ) and SwinUNETR (Hatamizadeh et al., ).

## 2 Related Work

### 2.1 Generative Models for Lesion Synthesis

Lesion synthesis using generative models has garnered significant attention for its potential to create diverse and realistic medical datasets, particularly in addressing the scarcity and imbalance of pathological data in biomedical applications. Early approaches have employed variational autoencoders (VAEs)(Kingma & Welling, ), generative adversarial networks (GANs)(Goodfellow et al., ), and more recently, diffusion models (Ho et al., ) to generate synthetic lesions across various medical imaging modalities and applications, including lung nodules (Han et al., ; Jin et al., ; Yang et al., ) and COVID-19 lesions (Lyu et al., ) in CT scans, colon polyps in colonoscopy (Shin et al., ; Pishva et al., ; Du et al., ), tumor cells in microscopy (Horvath et al., ), brain tumors in MRI (Billot et al., ), diabetic lesions in retinal images (Wang et al., ), and synthetic liver tumors (Lyu et al., ).

However, a persistent challenge for these methods is preserving anatomically accurate backgrounds alongside the lesions. In medical imaging, the background must respect the anatomical structure of the human body, which makes generating realistic backgrounds significantly more difficult than synthesizing isolated lesions, particularly in large-scale 3D images. While recent studies (Hamamci et al., ; Peng et al., ) have begun to address large-scale 3D medical image generation, these methods often require significant computational resources and extensive data.

Another limitation of current methods is the lack of explicit control when generating image-mask pairs. Typically, both the lesion and its corresponding mask are generated simultaneously, without explicit constraints linking the two. This results in limited control over key lesion properties, such as texture, size, location, and alignment between the image and mask. The absence of high-quality background preservation and fine control over these properties hinders the scalability and effectiveness, negatively impacting the performance of downstream tasks such as segmentation.

### 2.2 Background-Preserving Lesion Synthesis

In clinical practice, normal scans (either whole or partial) are far more abundant than pathological ones. For instance, in lung nodule cases, most pathological scans contain only a single lesion, yet traditional lesion synthesis methods often focus on small crops around the lesion, utilizing as little as $<1\%$(^1^11For example, a $64^{3}$ cube crop from a $512^{3}$ volume occupies $<0.2\%$ of the total voxels.) of the original data and leaving large portions of the normal background unused. This raises the question of whether it is necessary to rely on deep models to generate normal backgrounds.

Background-preserving lesion synthesis addresses this issue by starting from normal scans and synthesizing lesions through filling textures into manually generated lesion masks. This approach separates the generation of lesion masks and textures, allowing for finer control over lesions while maintaining the original background structure. Prior work has predominantly relied on heuristic-based methods (Yao et al., ; Hu et al., ), leveraging the abundance of normal backgrounds to generate lesions of varying sizes and textures at different locations. These studies have demonstrated that the generated data can significantly benefit downstream segmentation tasks.

However, these hand-crafted approaches rely heavily on manual adjustments to ensure that the generated lesions resemble real-world pathology. For example, Hu et al. () manually set grayscale values for texture synthesis, and lesion shapes are generated using morphological operations from ellipsoidal masks. Such hand-crafted rules limits the scalability and generalizability of these methods. In more complex scenarios, such as multi-peak and texture-rich lung nodules or multi-class cardiac lesions, these methods tend to fail (see Sec.4.2). This highlights the need for more flexible and robust data-driven techniques to address these challenges.

A recent study (Chen et al., ) uses conditional diffusion (Ho et al., ; Rombach et al., ) to synthesize abdominal tumors, as illustrated in Fig. 1a, building on background-preserving lesion synthesis . The image is first encoded with VQGAN (Esser et al., ), and a latent diffusion (Rombach et al., ) learns both the background and lesion, using the lesion mask and background (excluding the lesion) as conditional inputs. However, as the model still needs to generate background, the preservation of background integrity cannot be theoretically guaranteed. While it is possible to fill the area outside the mask with real background data, this may lead to inconsistencies in the final output. From our findings, this approach struggles with high-quality background generation, affecting downstream applications (Sec.4.2), particularly in data-limited scenarios. We also tested a similar image-space conditional diffusion model, which showed similar limitations, though image-space diffusion outperformed latent diffusion due to constraints imposed by the autoencoder.

A few concurrent studies (Lai et al., ; Wu et al., ; Zhu et al., ) have employed advanced generative models for lesion synthesis. Due to differences in research focus or/and the unavailability of their code/models, a comprehensive comparison could not be conducted. This study primarily focuses on diffusion-based approaches within the context of background-preserving lesion synthesis.

## 3 LeFusion: Lesion-Focused Diffusion Model

We propose LeFusion, a lesion-focused diffusion model that concentrates solely on lesion. In Sec.3.1, by combining forward-diffused backgrounds with reverse-diffused foregrounds, LeFusion reconstructs only the lesion, eliminating the need to model complex backgrounds. To address key challenges in lesion texture synthesis (Sec.3.2), LeFusion introduces histogram-based texture control for multi-peak lesions, allowing control over distinct lesion types, and a multi-channel decomposition strategy for multi-class lesions, where classes are generated in separate channels and combined into a single image. In Sec. 3.3, lesion mask diffusion enables control over lesion size, location and boundary, increasing mask diversity and enhancing lesion synthesis quality and flexibility.

### 3.1 Background-Preserving Generation via Inpainting

Figure: Figure 2: LeFusion: Lesion-Focused Diffusion Model. The top illustrates the training process of LeFusion, while the bottom shows the inference. During training, LeFusion avoids learning unnecessary background generation using a lesion-focused loss. In inference, by combining forward-diffused real backgrounds with reverse-diffused generated foregrounds, LeFusion ensures high-quality background generation. Additionally, we introduce histogram-based texture control to handle multi-peak lesions and multi-channel decomposition for multi-class lesions.
Refer to caption: x2.png

#### Decoupled Lesion and Background Generation.

Inspired by diffusion-based inpainting (Lugmayr et al., ; Avrahami et al., ), we aim to decouple the generation of lesions and background. As shown in Fig. 2, inpainting predicts the missing parts of an image, particularly lesions. For standard diffusion models (Sohl-Dickstein et al., ; Ho et al., ), the inference process starts by sampling a noise vector $x_{T}\sim\mathcal{N}(0,1)$ and gradually denoising it to produce the output image $x_{0}$. We focus on 3D images only in this study, while the method can be easily extended to 2D. The original grayscale image is denoted as $\hat{x}_{0}\in\mathbb{R}^{D\times H\times W\times 1}$, where $D$, $H$, and $W$ represent the 3D image size. The reverse diffusion step from $x_{t}$ to $x_{t-1}$ is decoupled into lesion and background components, as shown below. Here, $M_{f}$ and $M_{b}$ represent the lesion foreground mask and background mask. The lesion $o_{t-1}$ is predicted through the reverse diffusion process $p_{\theta}$ using a 3D U-Net, while $\hat{x}_{t-1}$ is derived from $\hat{x}_{0}$ by adding noise through forward diffusion $q$.

$$ ${x_{t-1}}=o_{t-1}\odot M_{f}+\hat{x}_{t-1}\odot M_{b}\\ ,o_{t-1}\sim p_{\theta}\left(\hat{x}_{t},t\right),\hat{x}_{t-1}\sim q\left( \hat{x}_{0},t\right)\\ .$ (1) $$

This approach preserves the background accurately without requiring prediction.

#### Making Diffusion Model Lesion-Focused.

The method above, while suitable for all diffusion-based inpainting, cannot guarantee that the denoised output $o_{t-1}$ is focused on the lesion area. Unlike standard diffusion training, where the target region may vary, we know that the part to be inpainted is specifically the lesion in our application. Therefore, we can design the model to predict only the lesion and ignore other regions. To achieve this, we introduce a lesion-focused loss during training. The general diffusion process starts with the original image $\hat{x}_{0}$, adding Gaussian noise over $T$ time steps (forward diffusion). The neural network is trained to predict the noise distribution at time step $t$ (reverse diffusion), conditioned on the noised image $\hat{x}_{t}$ and the time step. To ensure the model focuses only on the lesion, a mask $M_{f}$ is applied, calculating the loss exclusively within the lesion region. The training objective is defined as follows, where $\bm{\varepsilon}\in\mathbb{R}^{D\times H\times W\times 1}$ represents the noise sampled from a Gaussian distribution:

$$ $\mathbb{E}_{\hat{x}_{0},\epsilon\sim\mathcal{N}(0,1),t}\left[M_{f}\|\epsilon-p _{\theta}\left(\hat{x}_{t},t\right)\|_{2}\right]\\ .$ (2) $$

Despite the change in the training objective, inpainting inference remains unaffected. As shown in Eq. 1, outside $M_{f}$, the predicted lesion $o_{t-1}$ is replaced by the real noised background $\hat{x}_{t-1}$.

### 3.2 Fine Control of Lesion Textures

#### Handling Multi-Peak Distributed Lesions.

Figure: Figure 3: Illustration of Lung Nodule Texture Histogram Distribution. Samples are clustered into three groups based on the grayscale image histogram of lesions. The visualized differences between groups are significant, indicating a typical multi-peak distribution. These clusters roughly correspond to ground-glass, part-solid, and solid nodules.
Refer to caption: x3.png

In the section above, the model relies on the noised background $\hat{x}_{t}$ to infer the texture of lesion within the mask. While this works for lesions with minimal texture differences (as with cardiac lesions), it becomes problematic with multi-peak data. As shown in Fig. 3, lung nodules exhibit distinct texture types. We empirically show that relying solely on the background can lead to mode collapse.

To address this, we propose a simple yet effective approach, histogram-based texture control. The lesion texture histogram $h$ is used as a condition via cross attention Rombach et al. (), i.e.,

$$ $o_{t-1}\sim p_{\theta}\left(\hat{x}_{t},h,t\right)\\ .$ (3) $$

During training, the histogram is computed from the ground truth, and during inference, texture types can be controlled by adjusting the histogram.
Notably, this approach requires no additional lesion type annotations, such as nodule attenuation.

In Sec. 4, we demonstrate that the proposed histogram-based texture control is crucial for generating lesions on normal scans. Without it, models tend to fail, biasing towards healthy appearances and producing overly subtle lesions, which degrades the performance of downstream segmentation tasks using these synthetic data.

#### Joint Modeling of Multi-Class Lesions.

The method above focuses on single-class lesions, but many medical applications, such as cardiac MRI, require modeling multiple lesion types, like myocardial infarction (MI) and persistent microvascular obstruction (PMO). To capture correlations between lesion types and generate textures for multiple lesions simultaneously, we use a joint modeling strategy called multi-channel decomposition, where each channel corresponds to a different lesion type. The diffusion model generates each lesion in its respective channel, and they are combined through lesion masks.

We expand the input image $\hat{x}_{0}\in\mathbb{R}^{D\times H\times W\times 1}$ to $\hat{x}_{t}\in\mathbb{R}^{D\times H\times W\times n}$, based on the number of lesion classes $n$, where $n=2$ in the cardiac MRI experiments. Similarly, $\bm{\varepsilon}\in\mathbb{R}^{D\times H\times W\times n}$ and $o_{t-1}\sim p_{\theta}\left(\hat{x}_{t},t\right)\in\mathbb{R}^{D\times H\times
W
\times n}$ are extended to $n$ channels. The training objective is:

$$ $\mathbb{E}_{\hat{x}_{0},\epsilon\sim\mathcal{N}(0,1),t}\sum_{i=1}^{n}\left[M_{ f}^{(i)}\left\|\epsilon-p_{\theta}\left(\hat{x}_{t},t\right)\right\|_{2}^{(i)} \right]\\ ,$ (4) $$

where $*^{(i)}$ refers to the channel $i$ of $*$. To combine these channels, we compute $\sum_{i=1}^{n}\left[M_{f}^{(i)}o_{t-1}^{(i)}\right]$.

### 3.3 Diversifying Lesion Masks via DiffMask

Figure: Figure 4: DiffMask: Lesion Mask Diffusion. To achieve fine control over lesion size, location, and boundary, we propose two key designs: the boundary mask and the control sphere. The boundary mask removes areas outside the boundary at each diffusion step. The control sphere, trained using the bounding spheres of real masks, enables control over size and location during inference.
Refer to caption: x4.png

To further enhance the controllability and diversity of lesion generation, we introduce lesion mask diffusion (DiffMask). As shown in Fig. 4, to achieve fine control over lesion size, location, and boundary, we propose two key designs: the boundary mask and the control sphere. The former removes areas outside the boundary at each diffusion step, ensuring the generated mask stays within reasonable limits, while the latter manages the size and location of the lesion. During training, the control sphere is the bounding sphere of a real mask and is concatenated as a condition to the DiffMask input, with the real mask serving as the target. In inference, users can adjust the size and location to generate the desired lesion masks.

In terms of implementation, the architecture of DiffMask is similar to the texture diffusion model, also employing multi-channel decomposition to capture shape correlations and spatial distributions between multiple lesion masks. Each output channel is responsible for generating the lesion shape mask of a specific lesion. Finally, we apply a smoothing kernel as a post-processing step.

## 4 Experiments

### 4.1 Setup

#### Dataset.

LIDC: Multi-Peak Lung Nodule CT.

We use LIDC dataset (Armato III et al., ), which contains 1,010 chest CT scans, from which 2,624 regions of pathological (P) interest (ROIs) corresponding to lung nodules were extracted, along with 135 cases of healthy patients. The dataset was divided into an 808-case training set, comprising 2,104 lung nodule ROIs, and a 202-case test set, containing 520 lung nodule ROIs. Additionally, 3,076 normal (
N) ROIs were cropped from the 135 healthy patients, representing regions where lung nodules typically appear. These normal ROIs were used for data augmentation in the experiments.

Emidec: Multi-Class Cardiac Lesion MRI.
The Emidec dataset (Lalande et al., ) consists of examinations featuring DE-MRI in a short-axis orientation. This dataset offers access to 100 labeled cases, including 33 normal (N) and 67 pathological (P). The annotations cover 5 classes: background, left ventricle (LV), myocardium (Myo), myocardial infarction (MI), and persistent microvascular obstruction (PMO). We split the 67 P cases into 57 for training and 10 for testing. The 57 P cases are used to train the data synthesis model. In the downstream evaluation (Sec. 4.2), we use those models to synthesize P cases based on both 57 P and 33 N as the training set.

#### Method Comparison.

The following synthesis algorithms are compared with the LeFusion.

Copy-Paste.
We used the masks from real lesion data and matched them with normal data, copying the original lesion textures onto normal cases to generate new synthetic data.

Hand-Crafted (Hu et al., ). The lesion mask is represented by the overlapping of multiple ellipsoidal lesion masks, followed by several random morphological operations. The texture is approximated using Gaussian noise and softened through interpolation and Gaussian filtering.

RePaint (Lugmayr et al., ) or Blended-Diffusion Avrahami et al. (). This inference-stage method removes the lesion mask from the image and then fills in the lesion texture by combining forward-diffused backgrounds with reverse-diffused foregrounds. The model employs global training loss, which lacks the capability to focus on lesion information. Due to the absence of guidance from lesion category information, it is unable to specify corresponding lesions and is confined to simulating the generation of single-class lesions;

Cond-Diffusion (Ho et al., ; Rombach et al., ).

These methods use the lesion mask and background image information as conditional inputs (Rombach et al., ) to a diffusion model (Ho et al., ). However, a downside of this approach is that it disrupts the background information. Furthermore, directly using multiple masks as conditional inputs fails to control the corresponding categories, limiting the method to modeling the generation of single-class lesions.

Cond-Diffusion (L) (Chen et al., ).
Cond-Diffusion (L) is conceptually a latent diffusion (Rombach et al., ) version of Cond-Diffusion but adds VQGAN (Esser et al., ) to map image into latent space for diffusion. For a fair comparison, we fine-tuned open-source code and pre-trained weights by Chen et al. () and used the model outputs directly.

LeFusion and the Variants (Ours). Apart from standard LeFusion, there are two variants for fine control of lesion textures.
Histogram-Based Texture Control (*-H):
A variant of LeFusion that incorporates histogram control information, using the input histogram to guide the generation of multi-peak lesion textures.
Multi-Channel Decomposition (*-J):
When handling multi-class lesions, standard LeFusion trains individual models separately, lacking of modeling for the correlation between different lesions. LeFusion-J is a generalized version to model multi-class lesions jointly.

### 4.2 Improving Segmentation with Synthetic Data

#### Lung Nodule Segmentation.

We show that LeFusion can effectively benefit downstream application of training nnUNet (Isensee et al., ) and SwinUNETR (Hatamizadeh et al., ) to perform lung nodule segmentation. We use the following synthetic subset settings: P’: $2104\times 1$ synthetic ROIs from the 808 real pathological cases P; N’: $3076\times 1$ synthetic samples from the 135 normal subjects N; N”: $3076\times 2$ synthetic cases from the 135 normal subjects N.

Tab. 1 show the Dice and normalized surface distance (NSD). For the first group (Texture Synthesis with Real Masks), the texture of Hand-Crafted (Hu et al., ) and RePaint (Lugmayr et al., ) differs significantly from the real texture, making it challenging to achieve satisfactory results. Cond-Diffusion (Ho et al., ; Rombach et al., ) and Cond-Diffusion (L) (Chen et al., ), on the other hand, disrupt the background structure of the generated images. Our baseline model, LeFusion, is impacted by the pixel distribution of the background due to the lack of histogram control information, resulting in only a slight improvement over the baseline. We also synthesized lesion data on normal data N using Hand-Crafted Synthetic Masks and masks matched from lesion data, arriving at similar conclusions. In the final set of experiments, we validated the effectiveness of the DiffMask we designed for lesion synthesis and further explored accuracy improvements with increasing amounts of synthetic data. Compared to the baseline, in terms of Dice, we achieved improvements of 5.18% and 4.75% for nnUNet (Isensee et al., ) and SwinUNETR (Hatamizadeh et al., ), respectively. NSD improvements were 4.4% and 4.53%, respectively.

**Table 1: Downstream Lung Nodule Segmentation Dice ($\uparrow$) and NSD ($\uparrow$) on LIDC (Armato III et al., ). P: real pathological cases. P’/N’: synthetic pathological cases from pathological/normal cases. N”: more synthetic data than N’. Bold numbers indicate the best performance in each setting, with red highlighting significantly adverse effects compared to the baseline, and blue indicating significantly positive effects.**
| Methods | Training Setting | nnU-Net () | SwinUNETR () |  |  |
| --- | --- | --- | --- | --- | --- |
| Dice ($\uparrow$) | NSD ($\uparrow$) | Dice ($\uparrow$) | NSD ($\uparrow$) |  |  |
| Baseline | P | 78.26 | 88.90 | 78.38 | 88.67 |
| Texture Synthesis with Real Masks on P’ |  |  |  |  |  |
| Hand-Crafted () | P+P’ | 76.80 | 87.94 | 76.11 | 86.31 |
| Cond-Diffusion (; ) | P+P’ | 77.05 | 87.69 | 77.51 | 88.09 |
| Cond-Diffusion (L) () | P+P’ | 76.66 | 87.20 | 76.44 | 86.56 |
| RePaint () | P+P’ | 77.57 | 88.07 | 77.14 | 87.96 |
| LeFusion (Ours) | P+P’ | 78.77 | 89.25 | 78.43 | 88.54 |
| LeFusion-H (Ours) | P+P’ | 80.62 | 90.90 | 80.95 | 90.98 |
| Texture Synthesis with Hand-Crafted Synthetic Masks () on N’ |  |  |  |  |  |
| Hand-Crafted () | P+N’ | 75.10 | 85.50 | 74.88 | 84.64 |
| Cond-Diffusion (; ) | P+N’ | 76.62 | 86.44 | 76.66 | 87.20 |
| Cond-Diffusion (L) () | P+N’ | 76.71 | 86.83 | 77.20 | 87.88 |
| LeFusion (Ours) | P+N’ | 77.67 | 87.94 | 77.98 | 88.42 |
| LeFusion-H (Ours) | P+N’ | 80.19 | 89.75 | 80.08 | 90.42 |
| Texture Synthesis with Copied Masks () on N’ |  |  |  |  |  |
| Copy-Paste | P+N’ | 77.29 | 87.60 | 77.80 | 88.84 |
| Hand-Crafted () | P+N’ | 76.04 | 86.57 | 76.58 | 87.72 |
| Cond-Diffusion (; ) | P+N’ | 77.00 | 87.68 | 76.68 | 87.40 |
| Cond-Diffusion (L) () | P+N’ | 77.15 | 87.83 | 77.38 | 87.51 |
| LeFusion (Ours) | P+N’ | 78.49 | 89.22 | 78.55 | 89.06 |
| LeFusion-H (Ours) | P+N’ | 81.11 | 91.77 | 81.10 | 91.67 |
| Enhanced with Diffusion-Based Synthetic Mask (DiffMask) |  |  |  |  |  |
| LeFusion-H+DiffMask (Ours) | P+N’ | 82.66 | 92.49 | 82.63 | 92.77 |
| LeFusion-H+DiffMask (Ours) | P+N” | 83.19 | 93.21 | 83.07 | 93.10 |
| LeFusion-H+DiffMask (Ours) | P+P’+N” | 83.44 | 93.35 | 83.13 | 93.20 |

#### Cardiac Lesion Segmentation.

**Table 2: Downstream Cardiac Lesion Segmentation Dice ($\uparrow$) on Emidec (Lalande et al., ). The NSD metric is provided in Tab. A1. MI and PMO are [lesion classes. P: real pathological cases. P’/N’: synthetic pathological cases from pathological/normal cases. N”: more synthetic data than N’. Bold numbers indicate the best performance in each setting, with red highlighting significantly adverse effects compared to the baseline, and blue indicating significantly positive effects.**
| Methods | Training<br>Setting | nnU-Net () | SwinUNETR () |  |  |
| --- | --- | --- | --- | --- | --- |
| MI Dice ($\uparrow$) | PMO Dice ($\uparrow$) | MI Dice ($\uparrow$) | PMO Dice ($\uparrow$) |  |  |
| Baseline | P | 68.61 | 36.32 | 57.79 | 35.76 |
| Texture Synthesis with Real Masks on P’ |  |  |  |  |  |
| Hand-Crafted () | P+P’ | 69.60 | 36.06 | 57.64 | 34.96 |
| Cond-Diffusion (; ) | P+P’ | 66.89 | 37.76 | 56.75 | 36.31 |
| Cond-Diffusion (L) () | P+P’ | 68.07 | 31.93 | 56.97 | 32.72 |
| RePaint () | P+P’ | 69.14 | 28.93 | 55.14 | 33.86 |
| LeFusion (Ours) | P+P’ | 69.88 | 34.79 | 57.85 | 35.63 |
| LeFusion-J (Ours) | P+P’ | 69.95 | 38.01 | 59.61 | 37.99 |
| Texture Synthesis with Hand-Crafted Synthetic Masks () on N’ |  |  |  |  |  |
| Hand-Crafted () | P+N’ | 68.19 | 35.73 | 56.18 | 35.01 |
| Cond-Diffusion (; ) | P+N’ | 67.41 | 31.03 | 56.73 | 35.28 |
| Cond-Diffusion (L) () | P+N’ | 67.08 | 36.31 | 56.70 | 33.84 |
| LeFusion (Ours) | P+N’ | 69.17 | 37.18 | 59.42 | 34.83 |
| LeFusion-J (Ours) | P+N’ | 69.87 | 37.31 | 59.56 | 36.19 |
| Enhanced with Diffusion-Based Synthetic Mask (DiffMask) |  |  |  |  |  |
| LeFusion-J+DiffMask (Ours) | P+N’ | 69.81 | 40.62 | 58.94 | 39.00 |
| LeFusion-J+DiffMask (Ours) | P+N” | 70.17 | 42.44 | 58.60 | 41.24 |
| LeFusion-J+DiffMask (Ours) | P+P’+N” | 70.34 | 43.54 | 60.54 | 41.70 |
| LeFusion-J-H+DiffMask (Ours) | P+P’+N” | 71.28 | 43.41 | 59.30 | 42.49 |

We use the following synthetic subset settings :

P’: $57\times 1$ synthetic cases from the 57 real pathological cases P;
N’: $33\times 2$ synthetic cases from the 33 normal cases N;
N”: $33\times 5$ synthetic cases from the 33 normal cases N.

We use combinations of these subsets to train nnUNet (Isensee et al., ) and SwinUNETR (Hatamizadeh et al., ). The results of the two types of lesions are reported in 2 in terms of the Dice (0-100, higher is better). Due to page limitations, the corresponding NSD table is provided in Appendix Tab. A1.

In the first group of Tab. 2, we apply texture synthesis with real masks. The Hand-Crafted (Hu et al., ) produces textures that differ from real textures, leading to a decrease in baseline performance. Cond-Diffusion (Ho et al., ; Rombach et al., ) and Cond-Diffusion (L) (Chen et al., ) disrupts background structure, blurring lesion categories, decreasing MI’s performance. RePaint (Lugmayr et al., ), focusing on global information, struggles to generate textures that conform to lesion characteristics, resulting in a significant decrease in the Dice for PMO lesions. LeFusion models the two lesions separately, ignoring the correlation between lesions, which leads to improved accuracy for MI but decreased accuracy for PMO lesions. In contrast, our proposed LeFusion-J achieves superior results; For the second group, we expanded the normal data utilizing texture synthesis with hand-crafted synthetic masks. Due to RePaint (Lugmayr et al., )’s inability to distinguish between multiple lesion categories, we did not repeat experiments for it. From the experiments, we observed results similar to those of the first group; For the last, we evaluated our proposed lesion mask synthesis. Our method significantly improved the performance for both MI and PMO. Additionally, as data volume expanded, segmentation Dice consistently improved.

### 4.3 Visual Quality Assessment

Figure: Figure 5: Visualization of Synthetic Image on Emidec (Lalande et al., ) and LIDC (Armato III et al., ). We compare the differences in image similarity between synthetic pathological cases generated by different methods, using real pathological cases and using normal regions. More visualizations can be found in Appendix E
Refer to caption: x5.png

#### Image Quality.

Fig. 5 shows the generation results of different methods for lung nodule CT and cardiac MRI based on lesion images and normal images. We have also quantitatively calculated and compared the similarity between our generated images and real images; more details can be found in Appendix D. Fig. 5(a) displays the synthesized visualization of lung nodules (red). The Cond-Diffusion method (Ho et al., ; Rombach et al., ) and Cond-Diffusion (L) (Chen et al., ) disrupt the background structure. The lesions generated by Hand-Crafted (Hu et al., ) and RePaint (Lugmayr et al., ) fail to capture texture information, such as the grayscale variations characteristic of the lesions. Our baseline LeFusion, without histogram control information, is easily influenced by background features, resulting in the generation of relatively shallow lesions. In contrast, our LeFusion-H can better utilize histogram information to control the generation of lesions. Fig. 5(b) displays the synthesized visualization of two lesions for the heart: MI (blue) and PMO (red). The generation results for the heart show similar conditions. It is important to note that since the heart has two types of lesions, LeFusion-J (joint modeling of both lesions) can more accurately reflect the textures of both lesion types and, compared to LeFusion, results in smoother transitions at the boundaries between the two lesions and the background. However, due to the small contrast difference between heart lesions, LeFusion-J-H with histogram control achieves similar effects. More visualizations can be found in the supplementary materials.

#### Histogram Control Analysis.

We studied the effect of histogram control on the lung nodule dataset. The visualization results are shown in Fig. 6. Under different histogram controls, the attenuation (“transparency”) of the generated lesions changes from shallow to deep. Without histogram control information, the generated lung nodules tend to match the pixel distribution of the normal lung background, resulting in overly light lesions. We selected 100 subsets and used LeFusion-H and LeFusion to generate each sample twice, calculating the similarity between pairs using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), lower means more diverse. As shown in the figure, with histogram control, there is greater diversity between samples.

Figure: Figure 6: Illustration of Histograms Control Effectiveness. GT shows healthy lung tissue with the red area indicating the mask of the generated lesion. Control 1,2, and 3 are images generated under different histogram controls. Lower numbers indicate higher diversity in the generated images.
Refer to caption: x6.png

We also conducted a quantitative analysis of the impact of histogram control on the lesion area. The addition of the control histogram influences the distribution of the original data’s histogram, effectively shifting or weighting it based on the control input. Empirically, this interpretation aligns well with the observed distributions. Further details can be found in Appendix C.

#### Mask Quality.

We also visualize synthetic lesion masks, as shown in Fig. A1 and Fig. A2. Compared to the hand-crafted masks (Hu et al., ), our diffusion-generated masks are closer to the real masks and exhibit a more diverse range of shape patterns. More details can be found in Appendix A.

## 5 Conclusion

In conclusion, we introduce LeFusion, a novel lesion-focused diffusion model capable of recalibrating the diffusion learning objectives to lesion areas only. It preserves background by integrating forward-diffused background contexts into the reverse diffusion process. Our methodology is extended to handle challenging multi-peak and multi-class lesions, and further enhanced by a generative model for lesion masks, significantly diversifying our synthetic data. We demonstrate that synthetic data generated by our method can effectively boost the performance of state-of-the-art models like nnUNet (Isensee et al., ) and SwinUNETR (Hatamizadeh et al., ).

## Appendix A Mask visualization

Fig. A1 shows the real lung nodule lesion mask (a), the hand-crafted synthetic lung nodule mask (b), and the mask generated by our proposed diffusion model (d). Comparing subfigures (a), (b), and (d) in Fig. A1, it is evident that the real masks exhibit diverse shapes, while the DiffMask-generated masks closely resemble the real ones, also displaying varied forms with similar characteristics. In contrast, the handcrafted masks are relatively uniform in shape and differ significantly from the real masks.

Besides, our proposed DiffMask can control the size and location of the lesion masks, allowing for the generation of more diverse lesions. As shown in Fig. A1 (c) and Fig. A1 (d), we can use the sphere (Size Ball) to precisely control the desired lesion size and its position within the background image. This control enables us to produce various masks.

Figure: Figure A1: Visualization of Real / Synthetic Lesion Masks of Lung Nodule. (a) and (b) show the real masks and hand-crafted masks, (c) displays the control sphere (size ball) used by our proposed DiffMask, and (d) shows the corresponding synthetic mask results generated by DiffMask.
Refer to caption: x7.png

Fig. A2 shows that our diffusion-generated masks are closer to the real masks and exhibit a more diverse range of shape patterns, while Hand-Crafted masks consistently show large, continuous regions.

Figure: Figure A2: Visualization of Real / Synthetic Lesion Masks of Cardiac Lesion.
Refer to caption: x8.png

## Appendix B Downstream Segmentation Performance

**Table A1: Downstream Cardiac Lesion Segmentation NSD ($\uparrow$) on Emidec (Lalande et al., ). MI and PMO are lesion classes. P: real pathological cases. P’/N’: synthetic pathological cases from pathological/normal cases. N”: more synthetic data than N’. Bold numbers indicate the best performance in each setting, with red highlighting significantly adverse effects compared to the baseline, and blue indicating significantly positive effects.**
| Methods | Training<br>Setting | nnU-Net () | SwinUNETR () |  |  |
| --- | --- | --- | --- | --- | --- |
| MI NSD ($\uparrow$) | PMO NSD ($\uparrow$) | MI NSD ($\uparrow$) | PMO NSD ($\uparrow$) |  |  |
| Baseline | P | 59.27 | 29.19 | 47.66 | 20.51 |
| Texture Synthesis with Real Masks on P’ |  |  |  |  |  |
| Hand-Crafted () | P+P’ | 60.34 | 35.03 | 47.70 | 19.38 |
| Cond-Diffusion (; ) | P+P’ | 58.21 | 24.04 | 46.35 | 20.87 |
| Cond-Diffusion (L) () | P+P’ | 58.98 | 25.96 | 46.83 | 18.72 |
| RePaint () | P+P’ | 59.79 | 23.61 | 45.10 | 19.19 |
| (Ours) | P+P’ | 60.77 | 33.13 | 46.66 | 20.05 |
| LeFusion-J (Ours) | P+P’ | 60.44 | 36.65 | 48.23 | 24.11 |
| Texture Synthesis with Hand-Crafted Synthetic Masks () on N’ |  |  |  |  |  |
| Hand-Crafted () | P+N’ | 57.88 | 25.00 | 47.40 | 19.82 |
| Cond-Diffusion (; ) | P+N’ | 58.22 | 22.19 | 46.15 | 20.30 |
| Cond-Diffusion( L) () | P+P’ | 58.64 | 26.26 | 47.90 | 19.54 |
| LeFusion (Ours) | P+N’ | 60.75 | 23.96 | 47.81 | 19.91 |
| LeFusion-J (Ours) | P+N’ | 60.68 | 30.62 | 49.69 | 20.71 |
| Enhanced with Diffusion-Based Synthetic Mask (DiffMask) |  |  |  |  |  |
| LeFusion-J+DiffMask (Ours) | P+N’ | 61.35 | 38.93 | 48.62 | 22.43 |
| LeFusion-J+DiffMask (Ours) | P+N” | 61.48 | 35.03 | 50.00 | 23.92 |
| LeFusion-J+DiffMask (Ours) | P+P’+N” | 61.27 | 41.62 | 52.82 | 23.77 |
| LeFusion-J-H+DiffMask (Ours) | P+P’+N” | 62.74 | 40.96 | 50.73 | 24.25 |

Tab. A1, as a supplement to In Tab. 2, presents the NSD metric measured under the same experimental settings as in In Tab. 2.

In the first set of Tab.A1, we employ texture synthesis with real masks. The Hand-Crafted approach (Hu et al., ) generates textures that deviate from real ones, resulting in a decline in baseline performance. Cond-Diffusion (Ho et al., ; Rombach et al., ) and Cond-Diffusion(L)(Chen et al., ) disrupt the background structure, leading to blurring between lesion categories and reducing MI’s performance. RePaint (Lugmayr et al., ), which emphasizes global information, struggles to produce textures consistent with lesion characteristics, leading to a marked decrease in NSD for PMO lesions. LeFusion-S models the two lesions independently, disregarding the correlation between them, resulting in higher accuracy for MI but reduced accuracy for PMO lesions. Conversely, our proposed LeFusion-J outperforms the other methods; In the second group, we extended the normal data by synthesizing textures using hand-crafted synthetic masks. Due to RePaint (Lugmayr et al., )’s limitations in distinguishing multiple lesion categories, we did not conduct repeated experiments for it. The outcomes were similar to those of the first group; Lastly, we evaluated our proposed lesion mask synthesis. Our approach significantly enhanced the performance for both MI and PMO. Furthermore, as the data volume increased, the downstream segmentation NSD showed consistent improvement.

## Appendix C Histogram Control Analysis

For a given original image, the histogram information of its lesion region is represented as $I_{j}$, and the control information as $h_{1},h_{2},h_{3},\dots,h_{n}$ (where $i=1,2,3,\dots,n$), with $n$ being the number of controlled histograms. The resulting output image is defined as $O$, and $O$ can be derived as follows:

$$ $m_{i}=(r\times\log l_{j}+p)+(s\times\log h_{i}+q)$ (5) $$

$$ $O=e^{m_{i}},\quad i=1,2,3,\dots,n$ (6) $$

In this formula, $r$ and $s$ are scaling factors, and $p$ and $q$ are bias offset terms. Based on the above formulas, given the input image and corresponding control information, we can theoretically deduce the histogram of the lesion in the mask area of the generated image.

Fig. A3 shows the impact of histogram control information on the generation of lung nodules in a normal chest background. We randomly sampled 20 cases from the generated lesion data for statistical analysis. The first row represents the controlled histogram, where ”No control” indicates that the standard diffusion configuration without histogram control information was used. The second row shows the original mask areas of the 20 cases, as well as the average histogram effects of the generated and predicted data. Rows 3, 4, and 5 display the results of three different sample cases.

When the model does not use histogram control, the generated histograms of lung nodules in the normal background areas tend to be relatively shallow, influenced by the surrounding background. These histograms resemble the textures of the background mask areas, as shown in the rightmost column. In this column, the dark blue line represents the histogram of the image mask area generated by the diffusion model, while the orange line represents the theoretically fitted histogram control effect.

After introducing histogram control information, the interaction between the original image lesion areas and the controlled histogram effects causes the generated mask areas to shift increasingly to the right, following the peak effects of the three histograms (control 1, control 2, control 3). The second-row average histogram effect shows the stable trend of this shift.

Figure: Figure A3: Illustration of Histograms Control Effectiveness.The light blue histogram on the far left represents the original mask area. The pink histograms in the first row indicate the control information applied by the diffusion model. The yellow histogram represents the theoretically derived output lesion area, while the darker blue shows the actual generated lesion histogram.
Refer to caption: x9.png

## Appendix D Image Quality Evaluation

As shown in Tab. A2, we selected Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) to calculate the similarity between synthetic pathological cases generated by different methods and real pathological cases. As for cardiac lesions on the EMidec dataset, our proposed LeFusion achieved the highest average PSNR and SSIM. For lymph nodes with only one type of lesion on the LIDC dataset, LeFusion also achieved the highest PSNR. These experiments demonstrate that our synthesized lesions are more similar to real lesions across both CT and MRI modalities.

**Table A2: Synthesis Image Quality Assessment on Emidec (Lalande et al., ) and LIDC (Armato III et al., ). We compare the differences in image similarity between synthetic pathological cases generated by different methods given real pathological cases.**
| Methods | Emidec-MI | Emidec-PMO | Emidec-Avg. | LIDC |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ |  |
| Hand-Crafted () | 9.39 | 8.30 | 7.63 | 9.15 | 8.516 | 8.70 | 1.97 | 0.07 |
| Cond-Diffusion (; ) | 13.25 | 46.92 | 8.00 | 9.23 | 10.62 | 28.07 | 16.95 | 93.46 |
| Cond-Diffusion(L) () | 14.62 | 69.36 | 12.39 | 61.00 | 13.51 | 65.18 | 15.50 | 90.05 |
| RePaint () | 19.81 | 80.68 | 15.23 | 70.27 | 17.52 | 75.47 | 18.91 | 91.22 |
| LeFusion-S (Ours) | 25.65 | 91.78 | 27.71 | 89.42 | 26.68 | 90.60 | 22.38 | 90.16 |
| LeFusion-J (Ours) | 28.30 | 91.41 | 35.23 | 93.23 | 31.77 | 92.32 |  |  |

## Appendix E More Visualizations

In this section, we provide multiple illustrative figures demonstrating the effects of our proposed diffusion model, LeFusion.

Lung Nodule CT: Fig. A4 presents additional illustrations of the effects of histogram control. The histogram effectively controls the texture of the lesions, while the version without control information tends to generate lighter-colored lung nodules. Fig. A5 provides a visualization of the synthesized lesion results using real samples and their corresponding normal samples.

Figure: Figure A4: Illustration of Histograms Control Effectiveness. GT shows healthy lung tissue with the red area indicating the mask of the generated lesion. Control1, Control2, and Control3 are images generated under three different histogram controls.
Refer to caption: x10.png

Figure: Figure A5: Visualization of Synthetic Image on LIDC (Armato III et al., ). We compare the differences in image similarity between synthetic cases generated by different methods, using real pathological cases and normal regions.
Refer to caption: x11.png

Cardiac Lesion MRI: Fig. A6 shows the visualization of the denoising process at different stages in LeFusion for inpainting. Fig. A7 and Fig. A8
respectively show the generation of pathological results on lesion cases and normal cases.

Figure: Figure A6: Visualization of the Denoising Process in LeFusion for Inpainting. The process is conditioned on the masked input. Starting from a random Gaussian noise sample, the procedure iteratively denoises this input, progressively refining it into a high-quality image with the lesions.
Refer to caption: x12.png

Figure: Figure A7: Visualization of Synthetic Images Given Real Lesion Masks on Pathological Cases. Our LeFusion are compared with Hand-Crafted Hu et al. (), Cond-Diffusion Ho et al. (); Rombach et al. () and RePaint Lugmayr et al. ().
Refer to caption: x13.png

Figure: Figure A8: Visualization of Synthetic Images Given Synthetic Lesion Masks on Normal Cases. Our LeFusion are compared with Hand-Crafted Hu et al. (), Cond-Diffusion Ho et al. (); Rombach et al. () and RePaint Lugmayr et al. ().
Refer to caption: x14.png

## Appendix F Implementation Details

For the entire experiment, we used 6*A100 (40G) GPUs, including for diffusion and downstream segmentation tasks, with Python 3.8 and PyTorch version 2.4.0. Dataset: For lung nodules, we located each lesion and cropped and padded it to a size of 64x64x32. For the heart, we uniformly cropped and padded the size to 72x72x10. Diffusion Model: Training the LeFusion diffusion model for lung nodules requires approximately 3 A100 GPUs for one day. For inference, generating a single data takes about 30 seconds on one A100 GPU. Segmentation Model:
We implemented nnUNet (Isensee et al., ) and SwinUNETR (Hatamizadeh et al., ) using the MONAI framework. For downstream tasks, both SwinUNETR and nnUNet were trained for 200 epochs. Due to differences in dataset sizes, the training time on a single A100 GPU was approximately 6 to 24 hours for SwinUNETR and 4 to 10 hours for nnUNet.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please revise Figures 1 and 2 to more clearly illustrate the novelty of your proposed approach. Rather than emphasizing the strengths of the paper or incorporating numerous elements into a single pipeline, focus on presenting a straightforward and cohesive pipeline that highlights the mechanisms unique to your method.
soundness: 4
strengths: This manuscript is well-motivated, and the experimental results are satisfactory.
summary: This manuscript presents a diffusion model that utilizes forward-diffused backgrounds and reverse-diffused foregrounds as inputs, allowing the model to concentrate on reconstructing lesions specifically. Additionally, a post-processing method is applied to enhance generation quality.
weaknesses: There are several concerns regarding this manuscript:

* The novelty of the proposed approach is limited. The method does not significantly modify the underlying conditional diffusion process but instead introduces variations solely in the input.
* Figure 2 lacks clarity, and it would be beneficial to include the lesion-focused loss in this figure for a more comprehensive understanding.
* The writing lacks organization and is difficult to follow, which may impede readability and comprehension.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. In the tables (e.g. table 1), what do you mean by the significantly adverse/positive effects denoted by red/blue? Could you please clarify this in the text as well via a small note in the table caption(s)?
2. My suggestion: move image quality assessment quantitative results in the appendix (Table A2) to the main text if you have room. These are important metrics. You can shorten the related works to make space, that section doesn't need to be quite so extensive (or some of it could be moved to the supplementary).
    - Also, why didn't you evaluate unpaired perceptual metrics like FID, KID (https://arxiv.org/abs/1801.01401), SWD (https://arxiv.org/abs/1710.10196) etc.? the first two may have limitations for this task given that they use pretrained natural image features, but despite this they are still commonly used metrics for generative medical image models. I would consider adding these for future work, and also explaining why they are not used (particularly for the wider ICLR audience).
3. For the multiclass lesion case/-J model, did you study how performance/generation quality scales with adding more classes? This point may be a bit moot given how small the changes in performance were measured after adding the channel decomposition module to the base model, but I'm still curious.
soundness: 4
strengths: Major
1. The paper is polished, well-written and well-presented. Topics and concepts are organized and presented in a digestible fashion.
2. Overall, decent technical novelty. This incorporates many techniques which all come together to result in a strongly-performing methods, some pre-existing (such as combined noised backgrounds with denoised foregrounds), and some seemingly novel (such as histogram-based textural control). Also, despite the many components, the approach still seems relatively watertight because these additions are all pretty lightweight/simple (a good thing). No requirement for an additional network or something of that sort.
3. Overall, results are strong. Clear improvements over baseline methods is basically all cases, using reasonable metrics. They also study a range of training settings, which is good. Clear improvements over Cond-Diffusion, which would be the naïve approach that many would think of first trying for this task; the limitations of it as discussed in the introduction are clear from the experiments.
4. They also have fairly extensive ablation studies for their method, which is important given the number of components that they propose using. There are still a few related questions that I have, but they are minor.
5. In general, the evaluation is fair and appropriate. The datasets are challenging benchmarks, and I think two is sufficient given the wide range of experiments completed on them. There is also a good number of baseline models, especially considering that this task is relatively niche, so the methodological baselines that they compare to seem strong.

Minor
1. The motivation for this problem is clear: pathological subjects are indeed rare, especially for screening populations. Your survey of the limitations of existing lesion synthesis approaches also supports the motivation; for example, they result in low quality backgrounds, they lack precise control over generated lesions, etc.
2. The use of a histogram representation to condition the model on may seem too reductive for some applications, but it seems to work well here (makes sense given the clear correspondence between histogram shape/number of peaks and generated lesion morphology shown in Fig. 3), supported by the clear improvement to your method that including the -H module produced.
summary: The authors introduce a latent diffusion model-based method for inserting lesions into healthy medical images while also providing an accompanying mask. They utilize a number of additions to their model to address limitations of prior work or naïve approaches to this task (both pre-existing and seemingly novel), such as combining forward-diffused backgrounds with reverse-diffused foregrounds, introducing intensity histogram-conditioning to the diffusion model to control lesion texture, as well as techniques for further control of the shape, size etc. of the generated lesion. They evaluate their method for a variety of experimental scenarios on 3D cardiac MRI lesion and CT lung nodule generation, showing that their technique results in noticeable improvements to existing approaches with respect to using their generated data to train downstream task segmentation models.
weaknesses: Major
1. Some limitations of impact/scope: This task is clinically important but still fairly niche in medical image analysis, which itself is fairly niche within general machine learning and computer vision. The method (and task itself) also requires that dataset used needs the required annotations, which many medical datasets may not possess, and can be expensive/time-consuming to acquire. Overall, these limit the impact of the work somewhat, in the context of an ML conference at the level of ICLR, compared to a venue a bit more niche like MICCAI.

Minor
1. The benefits from using multi-channel decomposition (comparing the "-J" to no "-J" variants of your model in Table 2) are quite small. Can you provide some analysis or discussion of why this is the case, even if just hypothesizing? (However, I am guessing that the computational requirement to adding this component is practically negligible, so there is not really any harm in including it even if it results in only a very small performance improvement.)
2. You state in the abstract that synthesizing multi-peak and multi-class lesions is a "major challenge" I agree with the multi-peak case given how much your histogram-conditioning improved the generation of such lesions, but based on your channel decomposition module's only very small improvements to performance, I'm unsure if generating multi-class lesions could not already be done well by prior methods. Could you clarify this/point to your results that support this, and/or provide quantitative evidence that multi-class synthesis is challenging for prior approaches?

To summarize, the paper is methodologically solid, with some technical novelty, and demonstrates clear improvements to prior techniques for lesion generation tasks in medical images via well-designed experiments and baselines. However, the main limitation is just that the task is relatively niche within medical image ML, which makes it more niche within general ML, and so may be less impactful at a venue like ICLR as opposed to a medical imaging-focused venue such as MICCAI or MIDL. Still, these limitations do not take away the good things about the paper (of which there are many), so I vote for a marginal accept.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: For specific questions please refer to the points made in weaknesses.
soundness: 3
strengths: Lesion generating models are tools with significant potential for mitigating bias in medical vision AI algorithms concerning lesion detection, segmentation and quantification. Advancements in this topic should be highlighted in venues like this. 
The manuscript is sufficiently well written, all the provided Figures/Tables are insightful and adequately formatted. 
The choice of a 3D method for this inpainting problem is most adequate for CT and MRI. In these modalities, clinical lesion analysis workflows depend on the visualisation of multiple affected slices and 2D slice-wise inpainting methods would lead to slice-wise discontinuities. 

The proposed method is sufficiently contextualised in the Introduction and Related work sections, where the reseach gap is clearly defined. Beyond that, this gap is empirically demonstrated by experimenting with state-of-the-art approaches (Cond-Diffusion variants and RePaint). 

The proposed methodologies are thoroughly evaluated through comparisons with multiple other approaches, focusing on visual inspection of inpainted lesions (including comparison with real lesions) and their their downstream usability for training segmentation models. The latter evaluation used two different segmentation models, which contributes to the robustness of the findings across different segmentation training strategies. In addition, evaluating the approach on both MRI and CT datasets, ensures that the findings are not only applicable to one imaging domain. 

This paper provides multiple key contributions which not only address the research gap but also deal with modality specific challenges related to lesion texture and shape heterogeneity. The corresponding claims are well supported by the results.
summary: This paper introduces a novel 3D lesion inpainting method, LeFusion, which uses diffusion models to address data scarcity in medical imaging. Its primary aim is to generate synthetic lesions in lung CT and cardiac MRI scans for augmenting training data in lesion segmentation tasks. The approach is validated through both visual quality assessments and data augmentation derived segmentation 

performance improvement. Three key contributions can be summarised below: 
LeFusion Model: The authors identify that existing lesion inpainting methods struggle to preserve anatomically accurate backgrounds alongside the inpainted lesion, remarking that modelling the former is both hard and unnecessary. LeFusion is introduced to address this challenge incorporating two distinct features: (a) Training on a lesion focused diffusion loss, which only considers the lesion region. (b) Preserving the background at inference time with RePaint [1] by generating the lesion separately, while integrating forward-diffused background contexts into the reverse diffusion process. This design yields realistic lesions, better preserved backgrounds and improves data augmentation outcomes in both CT and MRI compared to non-lesion-specific models (Cond-Diffusion) both with and without RePaint based sampling. 

Modality-Specific Variants: Two specialized variants are introduced to address modality-specific challenges. LeFusion-H uses histogram-based conditioning to capture diverse lesion textures in CT, succesfully solving the texture mode collapse observed for the baseline LeFusion. LeFusion-J models multiple tissue subtypes in MRI via multi-channel decomposition, which enables the joint generation of different lesion tissue types typically observed in cardiac lesions. Both variants demonstrate superior data augmentation effectiveness in their respective modalities. 

DiffMask for Mask Generation: All variants of LeFusion rely on either existing real masks or handcrafted ones as priors for generating lesions in healthy scans. As a more flexible alternative, DiffMask is a diffusion model that generates synthetic lesion masks from basic spatial constraints, defined as a sphere with user specified location and size. Using the generated masks for data augmentation leads to the largest improvement in segmentation performance relative to the baseline in both CT and MRI.
weaknesses: While S4, the Introduction and Background sections seem to imply that the proposed lesion focused loss is a novel contribution proposed for the first time by the authors. This might not be necessarily true considering that there have been other works that employ similar approaches [2, 3]. While few and perhaps not as thoroughly evaluated, mentioning them could further strengthen the contextualisation of the approach. 

The description of the RePaint method in the experimental section implicitly suggests it consists of Cond-Diffusion using the RePaint [1] inference scheme. If that is the case it should be mentioned explicitly, if not then it should be better described. 
In the segmentation experiments, it is understood that masks priors for generating lesions in healthy scans (N’) are either derived from real masks, handcrafted or generated by DiffMask. However, additional information should be provided on how exactly the conditioning histograms in this N’ setting are selected when using LeFusion-H variants. 

Regarding DiffMask, the definition and role of boundary mask is not very clear. From Figure 4, it is presumed that it corresponds to the bounding box defining the volume crop centred on the lesion. However, the statement “The boundary mask removes areas outside the boundary at each diffusion step” challenges this concept. Further clarity on this point would be appreciated. Furthermore, it is only implicit, that the DiffMask takes the CT/MRI volume crop as an input in addition to the conditioning control sphere. Section 3.3. should be updated to enhance clarity on all these aspects. 

Adding supplementary details on how the model training and checkpoint selection was conducted for the RePaint, Cond-Diffusion, Cond-Diffusion (L) would improve transparency. 
 
[Minor]	 
More detail on the dataset preprocessing would be beneficial for further reproducibility. A mention to the volume resolution is particularly lacking. 

The choice of the specific crop-size could be further supported on previous work, for instance [4]. In addition, while not critical for acceptance, it would be interesting to study its effect over the results and would maybe answer the question: “How much local context is it necessary to generate realistic lesion?” 

While the purpose of the inpainted lesions is for downstream model training, further validating them using a radiologist would safeguard from potential biases that the generative model might be introducing the lesions. 

While describing Tables 1 and 2 it would be useful to clarify what is considered as “significant”. Since no standard deviations were provided, it is implied that these results were obtained for a single fold, so the concept of significance here is vague. In addition, while S5, the robustness of these findings to the specific data split could still be reinforced by adopting some sort of cross validation strategy. 
The authors left unclear whether the segmentation model was trained on the volume crops centred on the lesion or on the entire scans. From using the Copy-Paste method in the evaluation, the latter is presumed but it is not explicitly mentioned. 

In the cardiac MRI experiments, the LeFusion baseline of modelling the two lesion tissue types with separate models is mentioned as LeFusion in Table 2 but as LeFusion-S in Figure 5 and in the Appendix. It is suggested that the authors stick to one terminology. 
As a work mainly focusing on specific diffusion model mechanics for improved lesion inpainting, it makes sense that the evaluation focus on comparing different diffusion based methods. That said, it would still be interesting to see how GAN based approaches like [4, 5] would fair in this comparison. 
 
References:  
[1] Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. 
[2] Hansen, Colin, et al. "Inpainting Pathology in Lumbar Spine MRI with Latent Diffusion." arXiv preprint arXiv:2406.02477 (2024). 
[3] Rouzrokh, Pouria, et al. "Multitask brain tumor inpainting with diffusion models: A methodological report." arXiv preprint arXiv:2210.12113 (2022). 
[4] Yang, Jie, et al. "Class-aware adversarial lung nodule synthesis in CT images." 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019. 
[5] Wu, Linshan, et al. "FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis." arXiv preprint arXiv:2406.01264 (2024)

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. The reverse diffusion sampling process is not clearly defined; it appears to rely solely on the transformation in Equation (1), without detailing the sampling process or providing theoretical justification for omitting it.
2. Although the background from forward diffusion is used as the background in the reverse sampling process, and the loss constraint is applied only to the lesion area, how is continuity and smoothness ensured in the intersecting regions between the lesion and background?
soundness: 3
strengths: 1. The overall paper structure is clear and well-expressed.
2. A novel diffusion model is redesigned from the perspective of high-fidelity background preservation, with two texture generation control techniques developed to address multimodal and multiclass issues.
3. The comparative methods are recent benchmarks from the past two years, making the results highly convincing.
summary: This paper focuses on generating lesion-containing images from healthy images to address challenges in downstream segmentation tasks, such as real-world data scarcity and long-tail distribution issues. Previous research on medical image synthesis has primarily concentrated on lesion generation design, often overlooking high-fidelity background preservation. The authors propose a lesion-focused diffusion model, LeFusion, which maintains high-fidelity background by integrating the background from forward diffusion into the reverse diffusion process, thus simplifying the learning process and improving output control. Additionally, two effective strategies are introduced: histogram-based texture control and multi-channel decomposition to address the two main challenges in lesion texture synthesis: 1) multimodal and 2) multiclass lesions. The paper is well-written, with comprehensive experimental comparisons.
weaknesses: 1.There is a lack of detail on implementation specifics (such as the sampling process) and theoretical support for the method.
2. Analysis and discussion on the continuity at the fusion boundaries between lesion and background are missing, as well as the impact on downstream tasks.

Meta-review: This paper proposes LeFusion, a lesionfocused diffusion model. By redesigning the diffusion learning objectives to focus on lesion areas, the authors simplify the learning process while preserving high-fidelity backgrounds by integrating forward diffused background contexts into the reverse diffusion process. All reviewers agreed that the paper shall be accepted to this conference.

Decision: Accept (Spotlight)