# Diff-lung: diffusion-based texture synthesis for enhanced pathological tissue segmentation in lung CT scans

Rezkellah Noureddine Khiati, Pierre-Yves Brillet, Radu Ispas, Catalin Fetita

► To cite this version:

Rezkellah Noureddine Khiati, Pierre-Yves Brillet, Radu Ispas, Catalin Fetita. Diff-lung: diffusion-based texture synthesis for enhanced pathological tissue segmentation in lung CT scans. 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), Apr 2025, Houston, France. pp.1-5, 10.1109/ISBI60581.2025.10980827. hal-05236297

HAL Id: hal-05236297

https://hal.science/hal-05236297v1

Submitted on 2 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
DIFF-LUNG: DIFFUSION-BASED TEXTURE SYNTHESIS FOR ENHANCED PATHOLOGICAL TISSUE SEGMENTATION IN LUNG CT SCANS

Rezkellah Noureddine Khiati¹,³, Pierre-Yves Brillet², Radu Ispas³ and Catalin Fetita¹

¹SAMOVAR, Telecom SudParis, Institut Polytechnique de Paris, Evry, France

²Avicenne Hospital, AP-HP, Bobigny, France

³ Keyrus France, Levallois-Perret, France

ABSTRACT

Accurate quantification of the extent of lung pathological patterns (fibrosis, ground-glass opacity, emphysema, consolidation) is prerequisite for diagnosis and follow-up of interstitial lung diseases. However, segmentation is challenging due to the significant class imbalance between healthy and pathological tissues. This paper addresses this issue by leveraging a diffusion model for data augmentation applied during training an AI model. Our approach generates synthetic pathological tissue patches while preserving essential shape characteristics and intricate details specific to each tissue type. This method enhances the segmentation process by increasing the occurrence of underrepresented classes in the training data. We demonstrate that our diffusion-based augmentation technique improves segmentation accuracy across all pathological tissue types, particularly for the less common patterns. This advancement contributes to more reliable automated analysis of lung CT scans, potentially improving clinical decision-making and patient outcomes.

***Index Terms***— Pathological lung tissue, Segmentation, Diffusion based generation

1. INTRODUCTION

Interstitial lung diseases (ILDs) present a significant chal-
lenge in medical imaging due to the variability and sub-
tlety of pathological patterns involved (fibrosis, emphysema,
ground-glass opacities). These patterns are critical for diag-
nosing and monitoring various ILDs, yet annotated datasets
capturing these conditions are often limited, imbalanced, and
expensive to obtain. Although substantial work has been done
in the field, most studies have focused on segmenting individ-
ual pathologies, such as fibrosis [1, 2, 3] or emphysema [4, 5]
in isolation. Comprehensive segmentation that captures a
range of pathological textures within a single model remains
underexplored [6, 7]. This lack of a holistic approach limits
the generalizability of existing models in clinical practice,
where multiple pathologies often coexist and interact within
the lung tissue.

To address the limited and imbalanced datasets avail-
able for training, data augmentation techniques have gained
prominence, particularly in enhancing segmentation models
for medical imaging.

Generative adversarial networks (GANs) [8] have gained traction in this area, as they can produce realistic, augmented samples for medical imaging. However, GANs present challenges, including sensitivity to hyperparameters, instability during training, and risks of mode collapse [9], which can result in less diverse or overly homogeneous outputs. On the other hand, variational autoencoders (VAEs) [10] offer a more stable alternative, capable of generating conditioned images. Despite this, VAEs struggle to achieve high-resolution outputs with detailed textures, which are essential for accurate representation of lung pathologies.

Diffusion probabilistic models [11] offer a compelling solution to these challenges. Unlike GANs, diffusion models are less sensitive to hyperparameter tuning and are robust against mode collapse, while also achieving high-resolution generation. In this study, we propose a diffusion model guided by lung masks to synthesize various pathological textures across CT scans. This method enhances dataset diversity, addressing the class imbalance by generating realistic, synthetic samples to balance the representation of each pathology. These synthetic images will subsequently be used to train a baseline (UNet) [12] segmentation model, aimed at achieving comprehensive segmentation across multiple lung pathologies in a single framework.

Our contributions in this work lead to the development of
DiffLung, a diffusion model for generating lung images with
diverse pathological textures. To strengthen DiffLung, we
introduce a Class-Balanced Mask Ablated Training strategy
(CBMAT), which directs focused learning toward underrep-
resented pathologies, resulting in a balanced, robust dataset
for training a baseline segmentation model. We demonstrate
the effectiveness of the proposed augmentation strategy ver-
sus standard training of the same baseline model that still uses
optimized dataset class balancing.
## 2. METHOD

### 2.1. Diffusion models

Following recent advances in diffusion probabilistic models [11], we leverage their powerful capabilities for texture synthesis in medical imaging. Diffusion models define a Markov chain of diffusion steps that gradually convert a lung CT image $x_0 \in R^{c \times h \times w}$ into pure Gaussian noise $x_T$ through a fixed noise schedule. The process then learns to reverse this diffusion to generate new samples.

The forward diffusion process follows a variance schedule $\beta_1, \dots, \beta_T$ ($0 < \beta_t < 1$), which defines the noise level at each step. At each time step $t$, the process is defined as:

$$q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I) \quad (1)$$

This formulation means that at each step, we maintain a fraction $\sqrt{1-\beta_t}$ of the previous image while adding Gaussian noise with variance $\beta_t$. Eq. (1) becomes:

$$q(x_t|x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}x_0, (1-\bar{\alpha}_t)I) \quad (2)$$

where $\bar{\alpha}_t = \prod_{i=1}^t (1-\beta_i)$ represents the cumulative product of noise scheduling coefficients. The reverse process aims to gradually denoise the image by estimating $p(x_{t-1}|x_t)$. For this, a neural network $\epsilon_\theta$ is used to predict the noise component added during the forward process:

$$p_{\theta}(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_{\theta}(x_t, t), \sigma_t^2 I) \quad (3)$$

The mean of this distribution is computed as:

$$\mu_{\theta}(x_t, t) = \frac{1}{\sqrt{1-\beta_t}} \left( x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_{\theta}(x_t, t) \right) \quad (4)$$

where $\epsilon_\theta(x_t, t)$ is our noise prediction network that estimates the noise component in $x_t$. The network takes both the noisy image and the time step as input, allowing it to adapt its predictions based on the noise level.

The training objective is derived from variational inference, but can be simplified to a reweighted denoising score matching objective:

$$\mathcal{L} = E_{t \sim [1,T], x_0, \epsilon} [\|\epsilon_{\theta}(x_t, t) - \epsilon\|^2] \quad (5)$$

where $\epsilon \sim \mathcal{N}(0, I)$ is randomly sampled noise, and $x_t$ is obtained through the reparameterization trick:

$$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon \quad (6)$$

During sampling, we start from pure noise $x_T \sim \mathcal{N}(0, I)$ and iteratively apply the learned reverse process to obtain samples from $p(x_0)$. However, for our medical imaging application, sampling from the unconditional distribution $p(x_0)$ is insufficient. Following the mask-guided approach introduced in [13], rather than sampling from the unconditional distribution $p(x_0)$, the diffusion process can be controlled through

spatial anatomical guidance. This is achieved by conditioning the generation of a c-channel image $x_0 \in R^{c \times h \times w}$ on a multi-class anatomical mask $m \in \{0, \dots, C-1\}^{h \times w}$, where each value represents a specific tissue class among C possible classes (including background). This transforms the generative objective into sampling from the conditional distribution $p(x_0|m)$.

A key insight from this approach is that while the forward noising process $q(x_t|x_{t-1})$ remains unchanged since noise addition is independent of anatomical structures, the reverse process must be modified to account for the mask information. This modification affects both the reverse process, transforming from $p_{\theta}(x_{t-1}|x_t)$ to $p_{\theta}(x_{t-1}|x_t, m)$, and the noise prediction network $\epsilon_\theta$. Based on the evidence lower bound (ELBO) derivation for this conditional generation setup, the loss function becomes:

$$\mathcal{L}_m = E_{t \sim [1,T], (x_0, m), \epsilon} [\|\epsilon - \epsilon_\theta(x_t, t|m)\|^2] \quad (7)$$

where $\epsilon \sim \mathcal{N}(0, I)$ is randomly sampled noise, and the noise prediction network $\epsilon_\theta$ now takes the mask $m$ as an additional input to guide the denoising process.

### 2.2. Class-Balanced Mask Ablated Training (CBMAT)

In medical image analysis, mask-guided models often face the partial label problem, where missing annotations can mislead the model into treating unlabeled regions as background rather than potential pathologies. To address this challenge, [13] proposed a mask-ablated training (MAT) strategy which randomly removes mask classes with uniform probability (2$^{C-1}$ combinations). However, this uniform sampling fails to account for the inherent class imbalance in medical datasets. To overcome this, we propose a class-balanced mask ablated training (CBMAT) strategy that takes into account pathology prevalence in the training dataset. Specifically, we compute ablation probabilities inversely proportional to class frequencies:

$$P_{\text{initial}}(\text{ablation}|\text{class}_i) = 1 - f_i \quad (8)$$

where $f_i$ represents the frequency of class i in the training dataset. This formulation ensures that commonly occurring pathologies are ablated more frequently during training, forcing the model to learn robust representations of rarer conditions.

Additionally, to maintain anatomical consistency, we exclude the surrounding lung tissue class from ablation, ensuring the generated images retain correct structural boundaries. Furthermore, we incorporate a cosine annealing schedule for the ablation probabilities:

$$P_t(\text{ablation}) = P_{\text{initial}} \cdot \frac{1 + \cos(\pi t/T)}{2} \quad (9)$$
where *t* is the current training iteration and *T* is the total number of iterations. This scheduled approach gradually reduces the ablation frequency, allowing the model to first learn robust feature representations before fine-tuning on complete masks.

**Algorithm 1 Class Balanced Masked Ablation with Cosine Annealing**

**Require:** Number of mask classes *C*, dataset *p(x*<sub>0</sub>, *m*), class frequencies {*f*<sub>*c*</sub>}<sup>*C*−1</sup><sub>*c*=1</sub>, initial ablation probability *δ*<sub>0</sub>

**Ensure:** Model parameters $\theta$

1. Initialize $\delta_c \leftarrow \delta_0 \times (1 - f_c)$ for each class $c$ to inversely balance ablation based on frequency

2. **repeat**

3. Sample $(x_0, m) \sim p(x_0, m)$

4. **for** $c = 1, \dots, C - 1$ **do**

5. **if** $c$ is not the surrounding lung tissue **then**

6. Sample $\delta \sim \text{Uniform}(0, 1)$

7. **if** $\delta < \delta_c$ **then**

8. Set $m[m = c] \leftarrow 0$ {Ablate class $c$ if sampled}

9. **end if**

10. Update $\delta_c \leftarrow \delta_c \times 0.5 \times (1 + \cos(\pi \cdot \frac{\text{epoch}}{\text{max_epoch}}))$ 
{Cosine annealing for class ablation probability}

11. **end if**

12. **end for**

13. Sample $\epsilon \sim \mathcal{N}(0, I_n)$; $t \sim \text{Uniform}(\{1, \dots, T\})$

14. $x_t \leftarrow \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$

15. Update $\theta$ with $\nabla_{\theta} \|\epsilon - \epsilon_{\theta}(x_t, t|m)\|^2$

16. **until** converged

## 2.3. Mask Generation Process

The DiffLung model will be employed in data augmentation during training a baseline segmentation model. The objective is to compensate for underrepresented classes in the dataset by generating synthetic CT images with the desired spatial distribution of the disease lung patterns. In this respect, the available annotated lung masks are modified to include the required classes as follows. We compute the frequency of each pathology class within this dataset and then apply an iterative balancing algorithm which generates additional samples for less-represented classes. For each iteration, we identify regions belonging to the least frequent class and apply random augmentations such as rotation, copy-paste to a different location, and dilation. These transforms are applied with cascading probabilities, ensuring controlled variability. In order to avoid replacement of existing diseased regions in the lung mask, we constrain class augmentation only in the normal lung region. The process continues until the dataset reaches a threshold of class balance, enhancing the effectiveness of training for our segmentation model.

# 3. EXPERIMENTS

## 3.1. Dataset

The dataset for this study was collected at Avicenne Hospital, Bobigny, France and includes 156 patients with a total of 2266 CT slices. Of these, 137 patients (2076 slices) are used for training and validation and 19 patients (190 slices) for testing. The class distribution in the annotated lung masks is the following: 12.3% normal lung tissue, 0.3% emphysema, 3.8% interstitial lung disease (ILD, combining fibrosis and ground glass as a single entity), and 83.6% non-lung regions.

## 3.2. Generative model

We employed the segmentation-guided diffusion model [13] implementation, modifying the mask-ablated strategy with our Class-Balanced Mask Ablation Technique (CBMAT) to enhance focus on underrepresented classes. For preprocessing, we standardized the CT lung images to the Hounsfield Unit range of [-1000, 1000]. We isolated the surrounding tissue and lung regions using the corresponding masks, setting all other pixels to -1000 to prevent unnecessary reconstruction during denoising. We employed the DDIM scheduler [14] for faster inference with 1000 steps, and trained the model for 400 epochs.

## 3.3. Quantitative Results

We conducted a quantitative evaluation of our model, DiffLung with the CBMAT algorithm, comparing it against other segmentation-guided methods, including mask-ablated approaches and GAN-based architectures: Pix2Pix [15], a conditional generative adversarial network for image-to-image translation, and SPADE GAN [16], a semantic image synthesis method using spatially-adaptive normalization. The table compares the performance of different models using four key metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Floating Point Operations (FLOPS), and inference time. To ensure a standardized comparison, these metrics are normalized per image: time values are measured in seconds per image, and FLOPS are calculated in gigaflops (GFLOPS), representing $10^9$ floating-point operations. PSNR and SSIM are critical visual metrics that assess image quality. PSNR [17] quantifies reconstruction accuracy by measuring the signal's peak strength relative to noise, while SSIM evaluates the structural similarity between generated and ground truth images. Among the compared models, GAN-based architectures demonstrate the lowest inference times but compromise on visual quality metrics. In contrast, diffusion models, despite their computationally intensive iterative process, excel in producing high-quality outputs. Our proposed Diff-Lung approach outperforms the Seg-Guided-Diffusion model and the compared GAN architectures by effectively preserving structural details in gen-
erated images. While Diff-Lung and Seg-Guided-Diffusion share similar computational costs and inference times due to their common architectural foundation, our modified training strategy in CBMAT focuses on optimizing image quality. Although GAN-based models offer computational efficiency, our method prioritizes image quality—a trade-off particularly important in medical imaging applications.

<table><caption>**Table 1.** Comparison of models in terms of PSNR, SSIM, FLOPS, and Inference Time per Image.</caption><thead><tr><th>Models</th><th>PSNR ↑</th><th>SSIM ↑</th><th>FLOPS ↓</th><th>Time ↓</th></tr></thead><tbody><tr><td>Pix2Pix</td><td>17.56</td><td>0.61</td><td><strong>15</strong></td><td><strong>0.18</strong></td></tr><tr><td>SPADE</td><td>19.20</td><td>0.66</td><td>25</td><td>0.37</td></tr><tr><td>Seg-Guided-Diff</td><td>19.64</td><td>0.68</td><td>4000</td><td>21</td></tr><tr><td>Diff-Lung (CBMAT)</td><td><strong>20.70</strong></td><td><strong>0.71</strong></td><td>4000</td><td>21</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Qualitative comparison for pathological texture synthesis in lung CT images (Green: Healthy Tissue, Blue: Emphysema, Red: Fibrosis, Pink: thorax cage and mediastinum).</figcaption></figure>

## 3.4. Segmentation model

We have investigated the effectiveness of the proposed generative data augmentation approach by comparing the performance of a baseline segmentation model trained with and without synthetically augmented images. Taking into account the generative models results in Table 1, we also tested the performance gain when using DiffLung versus segmented-guided diffusion [13] as generative approach. We chose UNet as baseline architecture trained using CyclicLR scheduler

[18] with an exponential range and a Lookahead [19] wrapper optimizer, where SGD was used as the base optimizer to avoid local minima that could result from other classes especially affecting the underrepresented emphysema class. We trained three different models: 1) a reference model trained with classic data augmentation strategy - ref-UNet [6]; 2) a model trained with segmented-guided diffusion based data augmentation - Seg-Guided-UNet; and 3) a model trained with proposed DiffLung based data augmentation - DiffLung-UNet. The generative data augmentation used for the latter two models produced 534 additional slices, resulting in a class representation frequency of 6% for emphysema and 22% for fibrosis.

## 3.5. Results and discussion

The quantitative segmentation results of the baseline and generative-augmented models on the test database are synthesized in Table 2 in terms of Dice coefficients per lung tissue class. They show superior performance of the proposed DiffLung-based augmentation approach, also versus segmented-guided diffusion model which validates the effectiveness of the proposed data augmentation approach.

It is worth noting that the DiffLung model was trained using the same database as for the training the segmentation model. While this choice was imposed by the availability of the annotated data at our hand, we believe that the highly imbalanced lung tissue classes, even partially compensated by the CBMAT strategy, has limited the performance of the generative models (especially for emphysema, as seen in Fig. 1). Also, having unifying fibrosis and ground glass in a single ILD class will restrict the distinction of these patterns in the generated images. Future work will consider increasing the training dataset of the DiffLung model by including new samples of less represented classes, mainly emphysema, and separating the ILD class into fibrosis and ground glass.

<table><caption>**Table 2.** Quantitative comparison of Dice scores per class for different models.</caption><thead><tr><th>Models</th><th>D<sub>Healthy</sub> ↑</th><th>D<sub>Emphysema</sub> ↑</th><th>D<sub>Fibrosis</sub> ↑</th></tr></thead><tbody><tr><td>ref-UNet</td><td>0.91</td><td>0.60</td><td>0.72</td></tr><tr><td>Seg-Guided-UNet</td><td>0.90</td><td>0.73</td><td>0.76</td></tr><tr><td>DiffLung-UNet</td><td><strong>0.92</strong></td><td><strong>0.80</strong></td><td><strong>0.81</strong></td></tr></tbody></table>

## 4. CONCLUSION

This paper proposed DiffLung, a diffusion model for generating lung images with various diseased patterns exploiting a class-balanced mask ablated strategy to compensate for underrepresented classes in the training dataset. DiffLung demonstrated its ability to increase the performance of a lung texture segmentation model when used as generative data augmentation.
## 5. ACKNOWLEDGMENTS

This study was funded by the French National Research Agency (under the ANR MLQ-CT project).

## 6. REFERENCES

[1] Yuhan Ding, Weifang Xie, Kelvin K.L. Wong, and Zhifang Liao, "De-mri myocardial fibrosis segmentation and classification model based on multi-scale self-supervision and transformer," *Computer Methods and Programs in Biomedicine*, vol. 226, pp. 107049, 2022.

[2] Mujin Kim, Willmer Rafell Quiñones Robles, Young Sin Ko, Bryan Wong, Sol Lee, and Mun Yong Yi, "A predicted-loss based active learning approach for robust cancer pathology image analysis in the workplace," *BMC Medical Imaging*, vol. 24, no. 1, pp. 5, 2024.

[3] Sarah E. Gerard, Jacob Herrmann, Yi Xin, Kevin T. Martin, Emanuele Rezoagli, Davide Ippolito, Giacomo Bellani, Maurizio Cereda, Junfeng Guo, Eric A. Hoffman, David W. Kaczka, and Joseph M. Reinhardt, "Ct image segmentation for inflamed and fibrotic lungs using a multi-resolution convolutional neural network," *Scientific Reports*, vol. 11, no. 1, Jan. 2021.

[4] K. Ananthajothi, P Rajasekar, and M Amanullah, "Enhanced u-net-based segmentation and heuristically improved deep neural network for pulmonary emphysema diagnosis," *Sādhanā*, vol. 48, 02 2023.

[5] Plácido Gomes, Helder Bastos, André Carvalho, André Lobo, Alan Ranieri Guimarães, Walter Zin, and Alysson Carvalho, "Pulmonary emphysema regional distribution and extent assessed by chest computed tomography is associated with pulmonary function impairment in patients with copd," *Frontiers in Medicine*, vol. 8, pp. 705184, 09 2021.

[6] Simon Rennotte, Pierre-Yves Brillet, and Catalin Fetita, "Comparison of cnn architectures and training strategies for quantitative analysis of idiopathic interstitial pneumonia," in *Medical Imaging 2020: Computer-Aided Diagnosis*. SPIE, 2020, vol. 11314, pp. 57-69.

[7] Surendra Reddy Vinta, B. Lakshmi, M. Aruna Safali, and G. Sai Chaitanya Kumar, "Segmentation and classification of interstitial lung diseases based on hybrid deep learning network model," *IEEE Access*, vol. 12, pp. 50444-50458, 2024.

[8] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, "Generative adversarial networks," 2014.

[9] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin, "Which training methods for gans do actually converge?," 2018.

[10] Diederik P Kingma and Max Welling, "Auto-encoding variational bayes," 2022.

[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel, "Denoising diffusion probabilistic models," 2020.

[12] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation," 2015.

[13] Nicholas Konz, Yuwen Chen, Haoyu Dong, and Maciej A. Mazurowski, "Anatomically-controllable medical image generation with segmentation-guided diffusion models," in *International Conference on Medical Image Computing and Computer-Assisted Intervention*, 2024.

[14] Jiaming Song, Chenlin Meng, and Stefano Ermon, "Denoising diffusion implicit models," 2022.

[15] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros, "Image-to-image translation with conditional adversarial networks," 2018.

[16] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, "Semantic image synthesis with spatially-adaptive normalization," 2019.

[17] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," *IEEE Transactions on Image Processing*, vol. 13, no. 4, pp. 600-612, 2004.

[18] Leslie N. Smith, "Cyclical learning rates for training neural networks," 2017.

[19] Michael R. Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba, "Lookahead optimizer: k steps forward, 1 step back," 2019.