# Semi-supervised Brain Tumor Segmentation Using
Diffusion Models

Ahmed Alshenoudy, Bertram Sabrowsky-Hirsch, Stefan Thumfart, Michael

Giretzlehner, Erich Kobler

► To cite this version:

Ahmed Alshenoudy, Bertram Sabrowsky-Hirsch, Stefan Thumfart, Michael Giretzlehner, Erich Kobler.
Semi-supervised Brain Tumor Segmentation Using Diffusion Models. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.314-325, 10.1007/978-3-031-34111-3_27. hal-05449788

HAL Id: hal-05449788

https://inria.hal.science/hal-05449788v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Semi-supervised Brain Tumor Segmentation Using Diffusion Models

Ahmed Alshenoudy<fnref n="1" />(✉), Bertram Sabrowsky-Hirsch<fnref n="1" />(✉), Stefan Thumfart<fnref n="1" />(✉), Michael Giretzlehner<fnref n="1" />(✉), and Erich Kobler<fnref n="2" />(✉)

**Abstract.** Semi-supervised learning can be a promising approach in expediting the process of annotating medical images. In this paper, we use diffusion models to learn visual representations from multi-modal medical images in an unsupervised setting. These learned representations are then employed for the challenging downstream task of brain tumor segmentation. To avoid feature selection when using pixel-level classifiers, we propose fine-tuning the noise predictor network for semantic segmentation. We compare these methods against a supervised baseline over a varying number of training samples and evaluate their performance on a substantially larger test set. Our results show that, with less than 20 training samples, all methods outperform the supervised baseline across all tumor regions. Additionally, we present a practical use-case for patient-level tumor segmentation using limited supervision. The code we used and our trained diffusion model are publicly available (https://github.com/risc-mi/braintumor-ddpm).

**Keywords:** Denoising Diffusion Probabilistic Models · Medical Image

Segmentation · Few Shot Semantic Segmentation

## 1 Introduction

Medical image annotation is a very challenging, costly and time-consuming task that requires expert knowledge. Manual delineation for different regions of interest is still the standard practice, which is prone to errors and sometimes subject to inter- and intra-rater variability [7,26]. To expedite this annotation process, deep learning-based algorithms can be used, as they currently hold state-of-the-art performance on various medical image segmentation tasks [12]. However, these algorithms require large and representative datasets to achieve good performance, which is often impractical and tedious to acquire in the medical domain.

<footnote n="1">¹ Research Department Medical Informatics, RISC Software GmbH, Softwarepark 32a, 4232 Hagenberg, Austria</footnote>

<footnote>{ahmed.alshenoudy,bertram.sabrowsky-hirsch,stefan.thumfart, michael.giretzlehner}@risc-software.at</footnote>

<footnote n="2">² Department of Neuroradiology, University Hospital Bonn, Venusberg-Campus 1, 53127 Bonn, Germany</footnote>

<footnote>erich.kobler@ukbonn.de</footnote>
Therefore, creating a bottleneck for employing deep learning-based algorithms
in the medical domain where expert annotations are scarce.

Semi-supervised learning has emerged as a promising approach to address
this challenging issue. It attempts to tackle the practical situation where a sig-
nificant amount of unlabeled data is available in parallel with a few labeled
samples [20]. This is often the case in the medical domain, as thousands of
image acquisitions are carried out on patients daily and tight resources permit
only the labeling of a few exemplar annotations in contrast to annotating full
datasets. A common approach is to learn visual representations from the data in
an unsupervised setting and then learn a mapping between learned visual rep-
resentations and the available labeled samples. Different approaches have been
explored to learn visual representations, from constructing self-supervised tasks
[9,32] to using generative models [14,33]. With the recent success of diffusion
models [6] and their straightforward training, they are gaining more attention
and are explored in different application domains. Additionally, diffusion models
have also been used in unsupervised visual representation learning [4] and cur-
rently hold state-of-the-art performance for few-shot semantic segmentation on
multiple segmentation tasks.

In this paper, we leverage learned visual representations from diffusion mod-
els to tackle the challenging task of brain tumor segmentation. We also compare
the segmentation performance against a supervised baseline over a varying degree
of training samples. Instead of using a pixel-level classifier for the downstream
task, we propose fine-tuning the noise predictor network of the diffusion model.
Finally, we present a practical use-case, where we automatically annotate tumor
regions across different axial slices within the same patient. This is achieved by
using an exemplar tumor slice and two healthy tissue slices for training.

2 Related Work

Denoising Diffusion Probabilistic Models, or Diffusion models for brevity, are a class of likelihood-based generative models [10,18,27]. Dhariwal et al. [6] showed that they outperform state-of-the-art Generative Adversarial Networks (GANs) in terms of sample quality and diversity on multiple datasets. Because of this success, they have gained a lot of interest from the research community and have been explored in multiple applications such as super-resolution [25], in-painting [15], semantic segmentation [1,4,29], anomaly detection [21,28,31] and text-to-image generation [19]. A growing interest in using diffusion models for medical image analysis has also been observed, as outlined in a recent survey by Kazerouni et al. [13]. Other methods such as latent diffusion [23], inverse heat dissipation models [22] and blurring diffusion [11] have been motivated by denoising diffusion probabilistic models.

The core idea behind diffusion models is the transformation of a complex
data distribution into a simpler one, that is tractable through a defined forward
process. Thus, learning to approximate the reverse of this forward process would
define the distribution of the generative model [10,27]. For diffusion models,
this target data distribution is typically a standard Gaussian, which is achieved by a forward process that gradually adds noise over a fixed amount of T time steps. A trained diffusion model would transform noise $x_T \sim \mathcal{N}(0, I)$ to a sample $x_0$ by gradually denoising $x_T$. Additionally, diffusion models are latent variable models where latents $x_1, \dots, x_T$ share the same dimensionality as the original data $x_0 \sim q(x_0)$. The forward process is a Markov chain that gradually adds Gaussian noise according to a fixed variance schedule $\beta_1, \dots, \beta_T$:

$$q(x_t|x_{t-1}) := \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I), \qquad (1)$$

Forward sampling an input $x_0$ to any time-step $x_t$ can be done in closed form using $\alpha_t := 1 - \beta_t$ and $\bar{\alpha}_t := \prod_{s=1}^t \alpha_s$. The forward process can be rewritten as:

$$q(x_t|x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}x_0, (1-\bar{\alpha}_t)I), \qquad (2)$$

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, \quad \epsilon \sim \mathcal{N}(0, I). \qquad (3)$$

Diffusion models learn to approximate the reverse process:

$$p_{\theta}(x_{t-1}|x_t) := \mathcal{N}(x_{t-1}; \mu_{\theta}(x_t, t), \Sigma_{\theta}(x_t, t)), \qquad (4)$$

where in practice, this is done by predicting the noise component at each step $t$, using a noise predictor network $\epsilon_{\theta}(x_t, t)$, which is typically a U-Net architecture [24]. The mean of the distribution $\mu_{\theta}(x_t, t)$ in Eq. 4 is then calculated from a linear combination between predicted noise and $x_t$. For $\Sigma_{\theta}(x_t, t)$, it can be fixed to time-dependent constants or learned, where the latter was shown to improve likelihood-scores and allow for faster sampling [18]. Baranchuk et al. [4] showed that diffusion models are capable of learning visual representations and exploited these representations for semantic segmentation. Learned visual representations are acquired by forward sampling an image $x_0$ using Eq. 2 and extracting the intermediate activations of $\epsilon_{\theta}(x_t, t)$. These activations are then scaled to the original input size and are considered pixel-wise visual representations. For semantic segmentation, a Multi-layer Perceptron (MLP) is trained to classify these pixel-level visual representations to target classes.

Diffusion models and GANs have been previously explored for medical image segmentation. GANs have been used for segmenting 3D brain tumors by Cirillo et al. [5], who used it to generate segmentation maps, while also punishing the model for producing non-realistic segmentations. Diffusion models have been used for 2D brain tumor segmentation before by Wolleb et al. [29], where a diffusion model generates binary segmentation maps conditioned on the anatomical structure. Guo et al. [8] accelerated this by incorporating pre-segmentation results from another segmentation network. Wu et al. [30], added key improvements to address step-wise regional attention and the negative effects of high-frequency noise. Therefore, inspired by these works, specifically [4], we explore the use of diffusion models in a semi-supervised setting that replicates a practical scenario for medical image segmentation.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** An exemplar axial slice illustrating different modalities, tumor labels and regions. Tumor labels are shown for ED, ET and NCR in blue, green and yellow, respectively. Tumor regions are overlayed on different modalities, where from left to right are the WT, ET and TC. (Color figure online)</figcaption></figure>

# 3 Method

Our primary goal is to evaluate how learned visual representations from a diffusion model transfer to the downstream task of medical image segmentation. We designed an experiment that replicates a practical scenario, in which annotated data are scarce. The goal is learning to extend knowledge from a few annotated samples to the full dataset. We derived our method based on the approach presented by Baranchuk et al. [4] and tuned the hyper-parameters to our dataset as outlined in our published documentation.

## 3.1 Experimental Design

In our experiments, we employ data from the Brain Tumor Segmentation (BraTS) 2021 challenge [2,3,17]. The dataset contains multi-parametric Magnetic Resonance Imaging (mpMRI) scans from 2,040 patients, of which 1,251 are provided with ground-truth segmentations for algorithm development and training. For each patient, the mpMRI scan comprises four MRI modalities, which are T1-weighted (T1), post-contrast T1-weighted (T1Gd), T2-weighted (T2) and T2 Fluid Attenuated Inversion Recovery (T2-FLAIR). Each MRI volume has a size of 155 × 240 × 240, with uniform isotropic resolution (1 mm³). The ground-truth segmentations have three tumor labels, which are Necrotic Tumor Core (NCR), Peritumoral Edematous (ED) and Non-enhancing tumor (ET). For the challenge evaluation, metrics are reported based on different tumor regions. These regions are Whole Tumor (WT), which is a combination of all tumor labels, Tumor Core (TC), which results from ET ∪ NCR and the last region is the ET itself. Figure 1 illustrates the different MR sequences, tumor labels and regions.

We define a dataset $\mathcal{D} = \mathcal{D}_u \cup \mathcal{D}_p$, where $\mathcal{D}_u$ denotes the set of unlabeled samples and $\mathcal{D}_p$ denotes an available pool of annotated samples with a ratio to $\mathcal{D}_u$ of ≈ 9.5%. The entire dataset $\mathcal{D}$ was used to train a diffusion model for extracting visual representations. We then sample different training subsets $\mathcal{D}_n$ from $\mathcal{D}_p$, with varying training samples $n$, which we use to train different
downstream models and a supervised baseline. These models are then evaluated on the entire $D_u$. Additionally, we define an upper-bound to the task, where we trained a state-of-the-art supervised method on the entire $D_p$ and also evaluated on $D_u$.

For our experiments, we extract axial slices from the original 3D MRI volumes and the ground-truth segmentations. The longitudinal locations were stratified to increase the proportion of slices containing a segmented tumor: we chose seven locations to extract slices, starting from slice 70 up to slice 100, while skipping four slices in between. Axial slices are ordered in the superior direction through this paper. Afterwards, we normalized each extracted axial slice between [-1, 1] and down-sampled it to 128 x 128 using bilinear interpolation. Labels were down-sampled using nearest neighbors interpolation. This resulted in a dataset $D$ of 8,757 axial slices, which we split into a training pool and a test dataset. The training pool $D_p$ (757 slices), was used to randomly sample (without replacement) training datasets $D_n$ of varying size $n \in \{10, 20, 30, 40, 50\}$ for conducting different experiments. The test dataset $D_u$ comprised the remaining 8,000 slices, which we used to evaluate the different segmentation approaches in our experiments.

## 3.2 Diffusion Model Training

The training was based on the pipeline used in [18]. The model was trained for 360,000 iterations, with a learning rate of $2 \times 10^{-4}$, a batch size of eight images and an exponential moving average (EMA) over model weights of 0.995. We used a diffusion process with $T = 4,000$ steps and a cosine noise schedule, as both have been shown to boost model performance [18]. For simplicity, we kept $\Sigma_{\theta}(x_t, t)$ to untrained time-dependent constants. The $\epsilon_{\theta}(x_t, t)$, or $\epsilon$-predictor network, is based on a U-Net architecture, we use a down-sampling stack comprising six stages, where each stage has three residual blocks. The down-sampling stages had layer widths of $[C, C, 2C, 2C, 4C, 4C]$, where we set $C = 64$ and the up-sampling stack mirrored this down-sampling stack. We also use multi-head attention with four attention heads, however, applied only to resolutions of $32 \times 32$.

## 3.3 Semantic Segmentation Methods

**Pixel-Level Classifier.** We use the same multi-layer perceptron (MLP) architecture used in [4], which we call Simple MLP and another variant architecture which we name Deep MLP. These models learn to classify pixel-level visual representations to target classes. Visual representations are extracted from decoder layers $\{16, 17, 18\}$ after sampling an input to $t = 250$, where each layer had an output shape of $64 \times 64 \times 64$. Through empirical experiments, we found out that this configuration was a good choice for extracting visual representations for our downstream segmentation task. This configuration was founded on a visual analysis of learned visual representations, but also different configurations can be used. All pixel-level classifiers were trained for 6 epochs, with a batch size of 64 and a learning rate of $10^{-3}$.
**Noise Predictor Network.** Instead of extracting intermediate representations from the $\epsilon$-predictor network, we propose to fine-tune this network on the available labeled data and use it for prediction. This is possible, as the latent space of the diffusion model shares the same dimensionality as the original data. We train this network with two configurations, one with fixed weights in the encoder and the other with trainable weights. The final convolutional block was adjusted to the number of output classes. Model training was carried out on a batch of five images and a learning rate of $3 \times 10^{-3}$ for 150 epochs. We used a decay factor of 0.95 for the learning rate, employed whenever training loss plateaued and limited the learning rate to a minimum of $10^{-6}$. We incorporated random flipping, rotations, affine transformations and random adjustments of brightness and gamma to augment the images. Predictions were performed in the latent space, after sampling inputs to $t = 100$.

**Supervised Baseline and Upper-bound.** The nnU-Net architecture [12], currently holds state-of-the-art performance on various medical image segmentation tasks and was incorporated in the winning submission for BraTS 2021 challenge [16]. For training the nnU-Net architecture, we only use the 2D U-Net model as we are dealing with 2D axial slices. Models were trained with the default configuration, except for a few modifications. The overall training was restricted to 500 epochs instead of 1,000. Images were fed to the 2D U-Net in full resolution of 128 × 128 and post-processing of predictions was disabled. Models were trained directly on the entire data and not in the standard 5-fold cross-validation, except for the upper-bound model. The idea was to have a model-to-model comparison, as ensembling or post-processing can be applied to any model and would only skew the performance if not applied equally. Baseline models were trained on different $D_n$ and the upper-bound model was trained on $D_p$. Therefore, we provide an estimate for the upper and lower bounds of this task for the extracted dataset.

# 4 Results

Learned visual representations from our trained diffusion model are shown in Fig. 2. Earlier decoder layers capture high-level visual representations, which gradually increase in detail as we progress through different decoder layers. However, representations become noisier towards later decoder layers and as we sample to later time steps in the forward diffusion process, which aligns with the findings of Baranchuk et al. [4]. As our primary goal was to extract visual representations, we did not evaluate the generative quality of the model. In our downstream task evaluation, we include two commonly reported segmentation metrics, the Dice Similarity Coefficient (DSC) and Hausdorff Distance at the 95th percentile (HD95), besides we also report the standard deviation across different experiments. Results for our experiments are presented in Fig. 3, where we outline segmentation metrics for different methods while varying the number of training samples. For each $D_n$, results are plotted for five different experiments. Additionally, we report an upper-bound DSC of 79.36 ± 0.23, 71.76 ± 0.15, 74.85 ± 0.32 for WT, ET and
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Learned visual representations for sample input. Extracted from $\epsilon$-predictor network decoder layers {15, 16, . . . , 20, 22} and $t$ = {250, 2000, 4000} for BraTS_00489 axial slice 95. To visualize learned representations, we apply K-means clustering across different feature maps for each layer, with $K$ = 10.</figcaption></figure>

TC, respectively, while the upper-bound HD95 was 4.50 ± 0.10, 3.81 ± 0.07 and 3.21 ± 0.04 for WT, ET and TC, respectively. All methods surpassed the nnU-Net baseline in terms of DSC performance when only 10 samples were available for training. In terms of HD95 scores, the nnU-Net baseline and fine-tuned $\epsilon$-predictor network had an advantage over both pixel-level classifiers. This is evident for the Whole Tumor region, where the nnU-Net baseline and fine-tuned $\epsilon$-predictor network had consistently better HD95 scores. This difference can be attributed to pixel-level classifiers falsely predicting individual pixels, as they do not directly consider neighborhood information. As HD95 scores for ET and TC had a narrower gap between different methods, scores for WT region were better for the nnU-Net baseline and retrained $\epsilon$-predictor network because of better ED predictions. The performance gap narrows between all methods as more training samples are used, which we did expect. We detail the exact results for experiments with $D_{10}$ and $D_{50}$ in Tables 1 and 2. We show some exemplar predictions for all methods on three different cases in Fig. 4.

A practical use-case is shown in Fig. 5, in which we predict axial slices within the same 3D volume given only a few support slices. The support slices serve as exemplary segmentations to train the model, which then predicts the other axial slices. This could save time during the annotation process as correcting good estimates usually consumes less time than annotating from scratch. We used the Simple MLP architecture and trained on three sample slices, where only one
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Segmentation scores for different tumor regions against varying training sample counts. The top row shows Dice scores, while the bottom row shows Hausdorff Distance 95 percentile metric.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Prediction examples from different methods for training dataset $D_{10}$, showing axial slices for BraTS_00654, BraTS_00446 and BraTS_01393 from top to bottom and slices 95, 70 and 100, respectively. ED, ET and NCR are shown in blue, yellow and green, respectively.</figcaption></figure>

slice contained a tumor section and the other two healthy tissue. Representations were extracted from decoder layers {15, 16, 17, 18}, to include more high-level semantic features from layer 15. Training was done on slices {5, 60, 100} after sampling inputs to time-step $t = 100$. We evaluate predictions for slices
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Predicting axial slices for the full volume, after training with only three support slices. The top row illustrates exemplar axial slices from the T1Gd MR sequence, the middle row and the last row contain ground-truth segmentations and Simple MLP predictions, respectively.</figcaption></figure>

The {50, 70, 75, 80, 85, 90} and plot those predictions in Fig. 4. For axial slices that contain larger sections of the tumor, segmentation performance is excellent. However, for slices that the diffusion model was not trained on, which also comprise a very small area of pixels to segment, performance drops significantly. Segmentation metrics of these predicted axial slices are shown in Table 3.

<table><caption>**Table 1.** Segmentation scores on different metrics for different architectures trained on 10 samples.</caption><thead><tr><th rowspan="2">Architecture</th><th colspan="3">DSC</th><th colspan="3">HD95</th></tr><tr><th>WT</th><th>ET</th><th>TC</th><th>WT</th><th>ET</th><th>TC</th></tr></thead><tbody><tr><td>Simple MLP</td><td>69.20 ± 2.47</td><td>53.95 ± 6.75</td><td>56.32 ± 6.42</td><td>11.25 ± 1.99</td><td>5.07 ± 1.81</td><td>7.78 ± 2.21</td></tr><tr><td>Deep MLP</td><td><b>70.02 ± 1.54</b></td><td>53.47 ± 3.00</td><td><b>56.83 ± 1.41</b></td><td>11.77 ± 1.83</td><td><b>4.97 ± 0.95</b></td><td>7.24 ± 0.85</td></tr><tr><td>Retrained ϵ<sub>θ</sub></td><td>69.72 ± 1.24</td><td><b>54.78 ± 4.55</b></td><td>54.83 ± 4.33</td><td><b>7.89 ± 0.40</b></td><td>5.04 ± 0.93</td><td><b>6.33 ± 1.08</b></td></tr><tr><td>Baseline</td><td>65.83 ± 2.80</td><td>47.98 ± 7.42</td><td>50.73 ± 5.99</td><td>8.30 ± 1.29</td><td>5.89 ± 1.23</td><td>7.00 ± 1.14</td></tr></tbody></table>

<table><caption>**Table 2.** Segmentation scores on different metrics for different architectures trained on 50 samples.</caption><thead><tr><th rowspan="2">Architecture</th><th colspan="3">DSC</th><th colspan="3">HD95</th></tr><tr><th>WT</th><th>ET</th><th>TC</th><th>WT</th><th>ET</th><th>TC</th></tr></thead><tbody><tr><td>Simple MLP</td><td>74.70 ± 0.67</td><td>63.24 ± 0.94</td><td>67.34 ± 1.78</td><td>7.18 ± 0.47</td><td>3.50 ± 0.11</td><td>4.77 ± 0.55</td></tr><tr><td>Deep MLP</td><td>74.79 ± 1.43</td><td>62.76 ± 1.39</td><td>67.62 ± 1.31</td><td>7.27 ± 0.90</td><td>3.76 ± 0.24</td><td>4.87 ± 0.34</td></tr><tr><td>Retrained ϵ<sub>θ</sub></td><td><b>75.86 ± 0.84</b></td><td><b>66.19 ± 0.60</b></td><td><b>69.89 ± 1.25</b></td><td><b>5.39 ± 0.14</b></td><td><b>3.18 ± 0.14</b></td><td><b>4.09 ± 0.24</b></td></tr><tr><td>Baseline</td><td>73.66 ± 1.24</td><td>64.46 ± 1.84</td><td>68.75 ± 2.63</td><td>5.99 ± 0.41</td><td>3.37 ± 0.27</td><td>4.21 ± 0.43</td></tr></tbody></table>
<table><caption>**Table 3.** Segmentation metrics for exemplar axial slices within the same case BraTS_00000. Axial slices with larger tumor sections have excellent segmentation scores for both DSC and HD95.</caption><thead><tr><th rowspan="2">Axial Slice</th><th colspan="3">Dice</th><th colspan="3">HD95</th></tr><tr><th>WT</th><th>ET</th><th>TC</th><th>WT</th><th>ET</th><th>TC</th></tr></thead><tbody><tr><td>50</td><td>21.31</td><td>39.39</td><td>22.03</td><td>47.98</td><td>39.51</td><td>47.99</td></tr><tr><td>70</td><td>92.43</td><td>86.06</td><td>95.97</td><td>2.00</td><td>1.10</td><td>1.00</td></tr><tr><td>75</td><td>86.94</td><td>74.30</td><td>91.63</td><td>3.00</td><td>2.44</td><td>2.00</td></tr><tr><td>80</td><td>87.26</td><td>73.35</td><td>89.12</td><td>3.00</td><td>2.83</td><td>2.23</td></tr><tr><td>85</td><td>83.65</td><td>79.88</td><td>87.47</td><td>3.78</td><td>2.89</td><td>5.00</td></tr><tr><td>90</td><td>51.63</td><td>34.04</td><td>19.05</td><td>4.24</td><td>8.35</td><td>8.60</td></tr><tr><td>mean</td><td>70.54</td><td>64.50</td><td>67.54</td><td>10.67</td><td>9.52</td><td>11.38</td></tr></tbody></table>

# 5 Conclusion

We showed that diffusion models can be used for extracting visual representations from mpMRI data. We exploited these representations for the downstream task of semantic segmentation, utilizing different approaches. Pixel-level classifiers were used and we proposed fine-tuning the $\epsilon$-predictor network. This results in better HD95 scores in comparison to pixel-level classifiers. Fine-tuning is used to avoid selecting specific layers to extract visual representations. With limited training data, both pixel-level classifiers and $\epsilon$-predictor network performed much better than the nnU-Net baseline with fewer training samples. Furthermore, we illustrated a practical use-case, where a single axial slice with annotated tumor region and other slices with only healthy tissue are used to predict other tumor-containing axial slices with competitive segmentation performance. Given the current research progress in diffusion models, we expect to see even improved approaches based on our experiments and using alternative models such as blurring diffusion models. Our approach could be further improved by adapting a 3D architecture for the diffusion model and further adapting the fine-tuning approach of the $\epsilon$-predictor network. We suggest using our method in an active learning pipeline, such that most informative axial slices are sampled, annotated and then extended to the rest of the volume.

**Acknowledgments.** This project is financed by research subsidies granted by the government of Upper Austria. RISC Software GmbH is Member of UAR (Upper Austrian Research) Innovation Network.

# References

1. Amit, T., Shaharbany, T., Nachmani, E., Wolf, L.: Segdiff: Image segmentation with diffusion probabilistic models (2021)
2. Baid, U., et al.: The RSNA-ASNR-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification (2021)

3. Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data **4**, 170117 (2017)

4. Baranchuk, D., Voynov, A., Rubachev, I., Khrulkov, V., Babenko, A.: Label-efficient semantic segmentation with diffusion models. In: International Conference on Learning Representations (2022)

5. Cirillo, M.D., Abramian, D., Eklund, A.: Vox2Vox: 3D-GAN for brain tumour segmentation. In: Crimi, Alessandro, Bakas, Spyridon (eds.) BrainLes 2020. LNCS, vol. 12658, pp. 274–284. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72084-1_25

6. Dhariwal, P., Nichol, A.: Diffusion models beat GANS on image synthesis. In: Advances in Neural Information Processing Systems, vol. 34, pp. 8780–8794. Curran Associates, Inc. (2021)

7. Fiez, J.A., Damasio, H., Grabowski, T.J.: Lesion segmentation and manual warping to a reference brain: Intra- and interobserver reliability. Human Brain Mapping **9**(4), 192–211 (2000)

8. Guo, X., Yang, Y., Ye, C., Lu, S., Xiang, Y., Ma, T.: Accelerating diffusion models via pre-segmentation diffusion sampling for medical image segmentation (2022)

9. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000–16009 (2022)

10. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851. Curran Associates, Inc. (2020)

11. Hoogeboom, E., Salimans, T.: Blurring diffusion models (2022)

12. Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.: nnu-net: A self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods **18**, 203–211 (2021)

13. Kazerouni, A., et al.: Diffusion models for medical image analysis: A comprehensive survey (2022)

14. Li, D., Yang, J., Kreis, K., Torralba, A., Fidler, S.: Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8300–8311 (2021)

15. Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L.: Repaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11461–11471 (2022)

16. Luu, H.M., Park, S.H.: Extending nn-UNet for brain tumor segmentation. In: Brainlesion: Glioma. Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 173–186. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-09002-8_16

17. Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging **34**(10), 1993–2024 (2015)

18. Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 8162–8171. PMLR (2021)
19. Nichol, A.Q., et al.: GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In: Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 16784–16804. PMLR (2022)

20. Ouali, Y., Hudelot, C., Tami, M.: An overview of deep semi-supervised learning (2020)

21. Pinaya, W.H.L., et al.: Fast unsupervised brain anomaly detection and segmentation with diffusion models. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2022, pp. 705–714. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-16452-1_67

22. Rissanen, S., Heinonen, M., Solin, A.: Generative modelling with inverse heat dissipation (2022)

23. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695 (2022)

24. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

25. Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., Norouzi, M.: Image super-resolution via iterative refinement. IEEE Trans. Pattern Anal. Mach. Intell. 45(4), 4713–4726 (2023)

26. Sharp, G.C., et al.: Vision 20/20: Perspectives on automated image segmentation for radiotherapy. Med Phys. 41(5), 050901 (2014)

27. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 2256–2265. PMLR, Lille (2015)

28. Wolleb, J., Bieder, F., Sandkühler, R., Cattin, P.C.: Diffusion models for medical anomaly detection. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2022, pp. 35–45. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-16452-1_4

29. Wolleb, J., Sandkühler, R., Bieder, F., Valmaggia, P., Cattin, P.C.: Diffusion models for implicit image segmentation ensembles. In: Proceedings of the 5th International Conference on Medical Imaging with Deep Learning. Proceedings of Machine Learning Research, vol. 172, pp. 1336–1348. PMLR (2022)

30. Wu, J., et al.: Medsegdiff: Medical image segmentation with diffusion probabilistic model (2022)

31. Wyatt, J., Leach, A., Schmon, S.M., Willcocks, C.G.: Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 649–655 (2022)

32. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40

33. Zhang, Y., et al.: Datasetgan: Efficient labeled data factory with minimal human effort. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10145–10155 (2021)