Combining Reconstruction-based Unsupervised Anomaly Detection with Supervised Segmentation for Brain MRIs

Authors: finn  behrendt
debayan  bhattacharya
lennart  maack
julia  krüger
roland  opfer
alexander  schlaefer
d  behrendt
l m bhattacharya
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 13:31:44

Abstract: In contrast to supervised deep learning approaches, unsupervised anomaly detection (UAD) methods can be trained with healthy data only and do not require pixel-level annotations, enabling the identification of unseen pathologies. While this is promising for clinical screening tasks, reconstruction-based UAD methods fall short in segmentation accuracy compared to supervised models. Therefore, self-supervised UAD approaches have been proposed to improve segmentation accuracy. Typically, synthetic anomalies are used to train a segmentation network in a supervised fashion. However, this approach does not effectively generalize to real pathologies. We propose a framework combining reconstruction-based and self-supervised UAD methods to improve both segmentation performance for known anomalies and generalization to unknown pathologies. The framework includes an unsupervised diffusion model trained on healthy data to produce pseudo-healthy reconstructions and a supervised Unet trained to delineate anomalies from deviations between input- reconstruction pairs. Besides the effective use of synthetic training data, this framework allows for weakly-supervised training with small annotated data sets, generalizing to unseen pathologies. Our results show that with our approach, utilizing annotated data sets during training can substantially improve the segmentation performance for in-domain data while maintaining the generalizability of reconstruction-based approaches to pathologies unseen during training.

Main paper: 
 introduction: 
 Deep learning (DL) methods have advanced in their ability to detect and segment brain pathologies in MRI images (Lundervold and Lundervold, 2019) . However, acquiring annotated data for each pathology is a challenge, especially when considering screening tasks, where the objective is to detect any potential anomaly. Unsupervised anomaly detection (UAD) provides a potential solution by modeling the distribution of healthy brain MRI scans to identify anomalies as outliers. A common technique in UAD is reconstruction-based anomaly detection, where generative models (GM) are trained to reconstruct healthy brain images. At test time, the GMs fail to replicate pathologies, thereby revealing anomalies through discrepancies between input and reconstruction. This method only necessitates healthy data and enables the identification of pathologies not encountered during training, which poses a challenge for supervised models. However, the performance of reconstruction-based UAD methods is often surpassed by supervised models when sufficient task-specific data is available (Chen et al., 2020; Baur et al., 2021b) . Unlike supervised methods, UAD methods that rely on reconstructions do not directly learn the relationship between abnormal patterns and their corresponding annotations. Instead, the segmentation map is a byproduct of measuring the discrepancy between input and reconstruction. This results in a noisy anomaly map with potential false positives caused by the GM's imperfect reconstructions. Consequently, distinguishing actual anomalies from normal reconstruction errors can be challenging. An alternative approach is self-supervised UAD, where synthetic anomalies are introduced to the healthy brain images to train a segmentation network in a supervised manner. Unlike reconstruction-based UAD, this strategy produces distinct anomaly maps with high specificity, simplifying the discrimination of abnormal structures similar to the synthesized anomalies. However, the segmentation performance depends on the nature of the generated anomalies and tends to have limited generalization to real pathologies (Lagogiannis et al., 2023; Cai et al., 2023) . In this study, we aim to combine the strong generalization capabilities and high sensitivity of reconstruction-based methods with the high specificity of self-supervised methods. We develop a framework that employs a denoising diffusion probabilistic model (DDPM; DM) to generate pseudo-healthy reconstructions of potentially abnormal input images (reconstruction branch). Furthermore, an Unet is trained to segment anomalies based on the residual of the input and the pseudo-healthy reconstruction (segmentation branch). We consider different settings to obtain the annotations for the supervised training of the Unet. First, in the self-supervised setting, we introduce synthetically generated anomalies to healthy brain MRIs. Second, in the semi-supervised setting, we utilize a small amount of annotated data containing real pathologies. At test time, the unsupervised anomaly maps from the reconstruction branch and the supervised predictions from the segmentation branch are fused to a final anomaly score. The results demonstrate that in contrast to self-supervised methods, our approach allows to integrate supervision while maintaining the generalizability of the underlying reconstruction branch. Specifically, we can improve the Dice score of reconstruction-based UAD methods from 58.55 % to 69.68 % for tumors when using the same pathologies for training, while the Dice score for stroke lesions unseen during training increases from 24.74 % to 26.77 %. 
related work: 
 For reconstruction-based UAD, different architectures have been proposed as GM. While the majority focuses on Autoencoders (AE) (Baur et al., 2021a) or Variational autoencoders (VAE) (Zimmerer et al., 2019; Chen et al., 2020; Bercea et al., 2023a ,c), also vectorquantized VAEs (Pinaya et al., 2022) and GANs (Nguyen et al., 2021) have been employed. Moreover, it has been shown that utilizing denoising tasks for regularization with Unetlike AEs can improve the UAD performance (Kascenas et al., 2022 (Kascenas et al., , 2023)) . Consequently, DDPMs have emerged as a GM for reconstruction-based UAD (Wyatt et al., 2022; Behrendt et al., 2023a,b; Bercea et al., 2023b) . In self-supervised UAD, typically, synthetic anomalies are incorporated into normal brain images. Subsequently, Unets are trained to segment these synthetic anomalies (Tan et al., 2021 (Tan et al., , 2022;; Cho et al., 2022; Meissen et al., 2022a) . We note that while AE-based reconstruction methods may also fall under the category of self-supervised techniques, within this work, the term "self-supervised" refers to the aforementioned approach of training segmentation models using synthetic anomalies. Expanding on this strategy, DRAEM (Zavrtanik et al., 2021) employs a dual-network architecture comprising a generator and a segmentation network. The generator is trained to eliminate synthetic anomalies, thereby providing a pseudo-healthy reconstruction. The segmentation network is then used to segment the generated anomalies, given the concatenation of abnormal input and pseudo-healthy reconstruction. Note that for the generator network in DRAEM, inpainting of synthetic anomalies is enforced by calculating the reconstruction loss between reconstruction and the anomaly-free input. In contrast, in our approach, the reconstruction model is trained on healthy data in an unsupervised fashion to remove any abnormal structure that is not part of the healthy training distribution. Hence, we expect this approach to generalize more readily to real pathologies. The authors (Liu et al., 2022) take a similar approach, aiming to improve supervised segmentation performance by augmenting a dual-branch Unet with pseudo-healthy reconstructions. These reconstructions are generated by a Soft-Intro VAE trained on healthy data. In contrast, our proposed framework does not solely depend on supervised predictions. Instead, these predictions are combined with the unsupervised anomaly scores derived from reconstructions of a DM. We hypothesize that this combination enables general anomaly detection, particularly for pathologies unseen during training.  
method: 
 In this section, we introduce our framework for supervised anomaly detection with DMs (SADM), detailed schematically in Figure 1 .SADM integrates two primary branches: a DM for generating pseudo-healthy reconstructions (reconstruction branch) and a supervised Unet for segmentation (segmentation branch). We train SADM in two sequential stages.In the first stage, our objective is to train the DM to reconstruct healthy brain scans x = F DM θ (x) where x ∈ R H×W . The training of the DM focuses on optimizing parameters θ to minimize the l1-reconstruction loss:L Rec = |x -x|.(1)Stage II: Supervised SegmentationIn the second stage, the pseudo-healthy reconstruction generated by the DM trained in Stage I is utilized to support anomaly segmentation. Given an input scan with a real or synthetic anomaly ẋ ∈ R H×W and its corresponding ground truth annotation ẏ ∈ R H×W , we use the DM, trained in Stage I, to generate the pseudo-healthy reconstruction x = F DM θ ( ẋ). Next, we feed both the residual ( ẋx) and the original input ẋ into a Unet. After encoding both inputs, the resulting features are concatenated at each layer and fed to the Unet decoder to predict the segmentation map ŷ = F seg ϕ ( ẋx, ẋ). Only the Unet parameters ϕ are optimized to minimize the cross-entropy (CE) segmentation loss during Stage IIL Seg = CE( ŷ, ẏ).(2)The anomaly detection process leverages both components of our framework for anomaly segmentation. Given a potentially abnormal input ẋ, we generate a reconstruction x = F DM θ ( ẋ) by the DM. Next, we utilize F seg ϕ to derive the supervised anomaly prediction ŷ = F seg ϕ ( ẋ-x, ẋ). In addition, we utilize the pixel-wise structural similarity (SSIM (Wang et al., 2004) ) between input and reconstruction x = 1 -SSIM ( ẋx) for unsupervised anomaly scoring. The anomaly score (AS) is then derived as a combination of the unsupervised anomaly map and supervised anomaly prediction Anomaly Score (AS) = x + ŷ.(3)For pathologies similar to the anomalies seen during training, the supervised anomaly prediction will feature higher probabilities in abnormal regions, refining the unsupervised anomaly map. For unseen pathologies, the predicted probabilities are low such that x is unaltered. We hypothesize that this combination allows for comprehensive anomaly detection, leveraging the unsupervised anomaly map for general anomaly identification and the supervised prediction for precise segmentation of known abnormal patterns. 
data: 
 We use T1-weighted MRIs from the IXI data set to train the DM in Stage I. We separate a healthy test set consisting of 160 samples. The remaining data is partitioned into five training sets (N=358) and validation sets (N=44) for cross-validation. In Stage II, we utilize the strategy applied in (Zavrtanik et al., 2021) to generate pairs of synthetic anomalies and ground truth annotation based on the IXI data set (DRAEM). Additional information about the generation process and exemplary anomalies are provided in Appendix C. Additionally, for the weakly supervised setting, we utilize small subsets containing approximately 10% of the BraTS21 (BRATS, N=1251) (Baid et al., 2021; Bakas et al., 2017; Menze et al., 2014) , and ATLAS-v2.0 (ATLAS, N=655) (Liew et al., 2022) data sets. For evaluation, we utilize the remaining 1151 and 589 samples of the BRATS and ATLAS data sets, respectively. Furthermore, we utilize the augmented IXI test set (DRAEM) to assess the segmentation performance concerning synthetic anomalies.Pre-and post-processing: We resample all T1 MRI scans to a resolution of [1×1×1] mm and register them to the SRI24-Atlas (Rohlfing et al., 2010) . Subsequently, we perform skullstripping using HD-BET (Isensee et al., 2019) leading to volumes of size [192 × 192 × 160] voxels. Finally, we apply bias-field corrections, reduce the resolution by a factor of two and crop 15 top and bottom slices in the transverse plane. For post-processing, we apply median filtering with a kernel size of 5 to the unsupervised anomaly maps.We utilize DMs as GM within our proposed framework to generate pseudo-healthy reconstructions1 . Specifically, we use conditioned DDPMs (cDDPM) following the implementation of (Behrendt et al., 2023b) . For the supervised segmentation of the residual image, we utilize a Unet (Ronneberger et al., 2015) like architecture, adapted from (Kascenas et al., 2022) . The volumes are processed in a slice-wise fashion, sampling slices uniformly during training. At test time, we reconstruct the full volume by iterating over all slices. We compare our framework against different established baselines. We compare reconstructionbased AEs and VAEs (Baur et al., 2021a) , FAEs (Meissen et al., 2022b) , DDPMs (Wyatt et al., 2022) , pDDPMs (Behrendt et al., 2023a) and cDDPMs (Behrendt et al., 2023b) . Furthermore, we compare the feature-based reverse distillation method (RD) (Deng and Li, 2022) , the self-supervised Poisson image interpolation (PII) (Tan et al., 2021) and DRAEM-Net (Zavrtanik et al., 2021) approaches. Note that for PII we perform the anomaly generation based on the IXI data set. For all reconstruction-based methods, we utilize SSIM for anomaly scoring with a Gaussian kernel with standard deviation of σ ssim = 1, leading to a window size of k ssim = 9. Implementation details of our proposed framework and compared baselines are provided in Appendix B.  
experiments: 
 For all our experiments, we evaluate the BRATS and ATLAS data sets containing real pathologies and the IXI data set augmented with synthetic anomalies (DRAEM). We report the mean ± standard deviation across the different folds for the best possible Dice Score ([DICE]) as well as the Area under Precision-Recall Curve (AUPRC) to assess the segmentation performance. We evaluate different variants of SADM. In SADM res , the residual of input and reconstruction and the (abnormal) input are fed to the Unet, whereas in SADM, only the input is used. Furthermore, we consider Unet and Unet res , where, in contrast to SADM only the prediction of the Unet is used, ignoring the anomaly map of the unsupervised reconstruction branch. In Appendix D, we provide an ablation study on the weighted combination of the segmentation and reconstruction branch.We evaluate our approach in different settings. First, we assume the typical UAD case where only data with healthy labels is available. We use synthetic anomalies to obtain a supervised signal for the segmentation branch in SADM. We utilize the generation process proposed in DRAEM (Zavrtanik et al., 2021) to generate the anomalies. In this setting, we compare our framework to various UAD baselines. Results are reported in block I and block II of Table 1 . Across the compared UAD baselines in block I, cDDPMs show the highest Figure 2 : Examplary test cases for SADM res , trained and evaluated in the weaklysupervised setting with the BRATS and ATLAS data sets, respectively. For visualization purposes, we provide exemplary binary segmentation maps for the unsupervised anomaly score, the supervised prediction and the final AS, respectively. We derive the binarization threshold by optimizing for the best possible dice score.segmentation performance for real pathologies. Hence, we consider them as a reconstruction model for the SADM framework. For real pathologies, SADM res outperforms cDDPMs with performance improvements of 3.4 %, 12.3 % for the BRATS and ATLAS data sets, respectively. Considering the synthetic anomalies in the DRAEM data set, a substantially higher DICE of 76.72 % is reported for SADM res compared to the DICE of 11.94 % achieved by cDDPMs. Notably, while the DRAEM-Net shows relative performance improvements of 10.5 % over SADM res for synthetic anomalies, it fails to generalize to the real pathologies in the BRATS and ATLAS data sets. Even the Unet, trained with the same synthetic anomalies as in DRAEM-Net, outperforms DRAEM-Net considering real pathologies.Comparing SADM and SADM res , we observe that utilizing the residual of abnormal input and pseudo-healthy reconstruction in addition to the abnormal input substantially improves the segmentation performance across all data sets.In this section we investigate using our framework in a weakly-supervised setting. Instead of generating synthetic anomalies, we assume a small amount of annotated data is available and consider a subset of the BRATS and ATLAS data sets for training, respectively. We only train with one data set at a time to evaluate the generalization to unseen pathologies.The results for this weakly-supervised setting are reported in block III of Table 1 . Using a small subset of annotated data substantially improves the segmentation of all models when evaluating the same (in-domain) data set. However, the segmentation performance of Unet and Unet res is poor for data sets containing pathologies unseen during training. In contrast, both SADM and SADM res enhance the segmentation performance on in-domain data while maintaining or even improving the performance of unsupervised cDDPMs for unseen pathologies. A visualization of the anomaly maps coming from different branches of the SADM framework is provided in Figure 2 . 
discussion and conclusion: 
 A significant challenge of supervised methods that UAD addresses is the need for annotated training data. This is especially crucial when considering screening tasks where the type and shape of potential lesions are unknown. Therefore, it is highly desirable to achieve generalization to different kinds of lesions while minimizing false positive predictions. In this work, we aim for a framework that benefits from the robust generalization of reconstruction-based UAD methods and the high discriminative power of supervised strategies.Comparing the unsupervised and self-supervised approaches in Table 1 , the additional shape information typically improves the segmentation performance with the magnitude of improvement dependent on the lesion type. However, considering purely self-supervised models, it is evident that supervised training based on synthetic data can result in overfitting.In contrast, our proposed framework, improves the segmentation performance for anomalies of known shape and appearance while maintaining or even improving the generalization of reconstruction-based UAD for pathologies unseen during training. This indicates that the framework effectively utilizes the complementary information of the reconstruction and segmentation branches, as highlighted in Figure 2 . On the one hand, the supervised segmentation branch enhances the specificity for pathologies similar to the anomalies seen during training. On the other hand, the reconstruction branch maintains the high sensitivity of reconstruction-based UAD for any abnormal pattern unseen during the training of the DM. Furthermore, feeding the residual of input and reconstruction to the Unet in addition to the abnormal input can enhance the segmentation performance, particularly in the self-supervised setting. This indicates that the additional information in the residual may contribute to learning the deviation from a normal representation, potentially reducing the risk of overfitting to specific anomaly shapes. While the DRAEM-Net shares some similarities with our approach, there are significant differences. First, DRAEM-Net uses a generator network trained to remove synthesized anomalies. In contrast, our reconstruction branch employs a DM trained to reconstruct healthy data without explicitly enforcing the removal of specific anomalies. Second, instead of solely relying on the segmentation branch, we combine the supervised prediction with the unsupervised anomaly map derived from the reconstruction branch. As demonstrated in our experiments, these adaptations lead to improved segmentation performance and generalization, enabling the effective use of SADM in a weakly-supervised setting. Therefore, our framework adds a significant feature to UAD approaches, especially considering that some annotated data is typically available. In summary, our approach shows encouraging results, paving the way for a practical solution for UAD in brain MRI. Limitations are seen in the potential reconstruction of unhealthy structures by the reconstruction branch and in the investigated synthetic anomalies intended initially for industrial defect detection. Despite the demonstrated improvement in performance, we anticipate further enhancements when integrating more realistic synthetic anomalies. Additionally, we intend to include data sets featuring subtler anomalies or different imaging modalities to broaden the evaluation of our approach.  
 
 Appendix: Code available at https://github.com/FinnBehrendt/Supervised-Anomaly-Detection-with-Diffusion-Modelshttps://github.com/FinnBehrendt/Conditioned-Diffusion-Models-UAD 


Decision: The Meta-Reviewer has read all reviews and rebuttal of authors.

Reviewers have all confirmed they have taken the author rebuttal into account for their final recommendation.

There is a consensus among reviewers that the work is at the level required for publication and would make a positive addition to the conference.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel approach that integrates reconstruction-based and self-supervised anomaly detection techniques for brain MRIs. Initially, the method trains a diffusion model to reconstruct healthy brain images. Subsequently, these reconstructions are leveraged to train a Unet in a supervised manner for precise anomaly segmentation. Finally, by integrating both networks, the approach generates anomaly maps for test images. Through experimental evaluation, the method demonstrates advantages in detecting not only in-domain anomalies but also new unseen ones when compared to baselines.
main_review: main_review: The reviewer appreciates the authors' feedback and their additional efforts to refine their manuscript. They note that minor points have been clarified, contributing to the paper's overall coherence. The proposed methodology is interesting and well-presented. The reviewer thanks the authors and believes their contribution will be valuable for the conference.
strength_weakness: strength_weakness: - While not an actual weakness, it would be helpful to include an evaluation of a fully-supervised setting, such as training a supervised Unet on a larger subset of the BRATS/ATLAS/DRAEM datasets and then evaluating it on the rest of them. This comparison would allow for an assessment against the presented methods designed for minimal annotation settings, quantifying their gap to the supervised scenario.
questions: questions: The paper is well-organized, justifying design choices through intuition and experimental results. A brief consideration of the minor suggestions outlined above could add to its completeness.
paper_summary: paper_summary: In this paper, the authors build on a previously proposed framework for UAD in industrial image (DRAEM), which they modify in several  ways to produce a novel method, adapted to unsupervised anomaly detection (UAD) or weakly supervised anomaly detection. The authors design several experiments, performed on multiple public datasets, to prove the superiority of their proposed method.
main_review: main_review: The reviewer thanks the authors for the comments and insightful discussions. The reviewer thinks the highlighted weaknesses were addressed by the authors and will keep his rating.
The reviewer still disagree with "self-supervised methods involve introducing synthetic anomalies and training a segmentation network to predict the anomaly's location", as the reviewer believes that self-supervised methods are much larger than this, even when restricted to UAD. However this is a very minor comment as the authors have precised what they meant in the manuscript.
Thanks again for the responses and discussion.
strength_weakness: strength_weakness: 1) The anomalies detected in both testing sets seem big and obvious to see, suggesting that a more challenging task would reinforce the findings of this study.
2) The reviewer believes both the noise used for the DDPM and the synthetic anomalies used for supervised segmentation are quite fitted (same statistics, size and intensity) to the anomalies tested for detection, which could hinder the performances for detection of other, different, types of anomalies.
questions: questions: - The reviewer believes the term "self-supervised", as used here, would have to be precised or changed. For instance using an auto-encoder reconstruction error (for training and testing) is considered self-supervised learning. The reviewer does not understand why the authors differentiate between self-supervised methods and reconstruction methods. Indeed the proposed method is fully unsupervised (or weakly-supervised), despite using supervised training, which can be difficult to grasp at a first reading, but this does not justify in the reviewer's point of view the use of "self-supervised" wording.
- Can the authors briefly comment on the use of l1 reconstruction loss instead of l2 ?
- Can the authors precise the window size used for SSIM ?
- Can the authors comment on why it is necessary to register the volumes to a common atlas ?
- The authors used the sum of the unsupervised anomaly map (x_tilde) and supervised anomaly prediction (y_dot_hat) as anomaly score map. The reviewer would be very curious to see an additional experiment, perhaps in appendix, of any metric, say Dice, plotted against a parameter $\Beta$, where the score map used would be  $\Beta$ x_tilde + (1-$\Beta$)y_dot_hat. Visualizing the peak and more generally the profile of such a curve would allow to see the importance and usefulness of each of the score maps used.
- The reviewer would like the authors to comment the fact that SADM_residual has rhoughly the same performances as SADM_input for the weakly supervised setting. Furthermore, the reviewer believes the sentence "Furthermore, feeding the residual of input and reconstruction to the Unet (SADMresidual )instead of the abnormal input only (SADMinput ) enhances the segmentation performance." has to be nuanced in light of the previous comment.
- Although common in UAD for medimage, the reviewer would appreciate if the authors can comment on the fact that the anomalies detected are still quite big and obvious to see (ATLAS and BraTS), and as a perspective refer to more challenging UAD tasks.
paper_summary: paper_summary: The authors propose a framework which combines reconstruction-based and self-supervised unsupervised anomaly detection. In the first training stage, pseudo-healthy reconstructions from an unsupervised diffusion model are learned. In the second stage, a segmentation network (Unet) is trained in a self-supervised utilising the generated pseudo-healthy images or in a semi-supervised way.
main_review: main_review: The reviewer appreciates the authors' responsiveness to their feedback. With revisions addressing the reviewer's comments, the manuscript has shown improvement, leading the reviewer to upgrade their final rating to a 'weak accept.' In future submissions, the reviewer suggests expanding the evaluation beyond brain datasets to encompass less invariant data, as this could enhance the relevance and appeal of the method, e.g., regarding robustness.
strength_weakness: strength_weakness: - The authors test the proposed method on MRI Brain datasets, especially, training in an unsupervised or self-supervised (PII and DREAM) way on the IXI dataset. Unhealthy data is generated synthetically or taken from other datasets like ATLAS and BRATS. Brains are less variant than other body parts. A comparison, e.g., with Chest or Full-Body datasets would have been interesting and more convincing regarding the generalisability of the proposed method.

- The proposed method is evaluated against AE-based methods and Diffusion models. There was no comparison with SOTA Diffusion-model-based unsupervised/weakly supervised anomaly detection methods, e.g., [1,2].

- PII produces as in its original implementation only subtle synthetic anomalies which almost vanish in noise. The anomalies generated following Zavrtanik's DREAM framework are also less natural. A comparison with, e.g., naive hypo/hyper-intense synthetic anomalies would have been interesting. Also, a comparison with other SOTA Self-supervised methods, e.g., [3,4] would have been more convincing.

[1] Wolleb, J., Bieder, F., Sandkühler, R., & Cattin, P. C. (2022, September). Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention (pp. 35-45). Cham: Springer Nature Switzerland
[2] Pinaya, W. H., Graham, M. S., Gray, R., Da Costa, P. F., Tudosiu, P. D., Wright, P., ... & Cardoso, M. J. (2022, September). Fast unsupervised brain anomaly detection and segmentation with diffusion models. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 705-714). Cham: Springer Nature Switzerland.
[3] Schlüter, H. M., Tan, J., Hou, B., & Kainz, B. (2022, October). Natural synthetic anomalies for self-supervised anomaly detection and localization. In European Conference on Computer Vision (pp. 474-489). Cham: Springer Nature Switzerland.
[4] Baugh, M., Tan, J., Müller, J. P., Dombrowski, M., Batten, J., & Kainz, B. (2023, October). Many tasks make light work: Learning to localise medical anomalies from multiple synthetic tasks. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 162-172). Cham: Springer Nature Switzerland.
questions: questions: - This sentence is not fully clear: "Instead of learning the removal of synthetic anomalies during training, our model learns to remove any abnormal structure that is not part of the healthy training distribution." How do these learning strategies differ?

- Were there any experiments made with 3D instead of slice-wise training, and other imaging modalities?

- For post-processing, the authors apply median filtering with a kernel size of 5 to the generated anomaly maps. How does this influence the performance regarding tiny/small anomalies?

- How much training time was needed for training stage 1+2? Is it possible to merge both stages into one?

Comments: title: Response to reviewer V5u5
Response to reviewer V5u5
Response to reviewer ukmk
Response to reviewer sxA9
Response to all reviewers
comment: The reviewer thanks the authors for the comments and insightful discussions. The reviewer thinks the highlighted weaknesses were addressed by the authors and will keep his rating.
The reviewer still disagree with "self-supervised methods involve introducing synthetic anomalies and training a segmentation network to predict the anomaly's location", as the reviewer believes that self-supervised methods are much larger than this, even when restricted to UAD. However this is a very minor comment as the authors have precised what they meant in the manuscript.
Thanks again for the responses and discussion.
Detailed Comments:

1.	"In stage II, the parameters ϕ are fixed" : the reviewer believe the authors meant \theta

We thank the reviewer for identifying this and revised manuscript accordingly.

2.	Can the authors precise the term "FAE" used to describe Meissen 2022b work ? At a quick glance the reviewer did not find correspondance to such an acronym in Meissen's work.

The reviewer is right. There is no such term in the cited work of Meissen 2022b. Unfortunately, we referred to a different work of the same authors. The right reference would be:
[Meissen, F., Paetzold, J., Kaissis, G., Rueckert, D., 2022b. Unsupervised anomaly localization with structural feature-autoencoders. arXiv preprint arXiv:2208.10992].
We apologize for the confusion. We corrected the reference in the revised manuscript.
1.	The reviewer believes the term "self-supervised", as used here, would have to be precised or changed. For instance using an auto-encoder reconstruction error (for training and testing) is considered self-supervised learning. The reviewer does not understand why the authors differentiate between self-supervised methods and reconstruction methods. Indeed the proposed method is fully unsupervised (or weakly-supervised), despite using supervised training, which can be difficult to grasp at a first reading, but this does not justify in the reviewer's point of view the use of "self-supervised" wording.

We agree with the reviewer that the terms “self-supervised” and “reconstruction-based methods” are overlapping. From our understanding of the UAD literature, there seems to be an established naming convention, though: Reconstruction-based approaches typically rely on the disparity between the input and its reconstruction for anomaly detection. On the other hand, self-supervised methods involve introducing synthetic anomalies and training a segmentation network to predict the anomaly's location. We clarified the naming in Section 2: “We note that while AE-based reconstruction methods may also fall under the category of self-supervised techniques, within this work, the term "self-supervised" refers to the aforementioned approach of training segmentation models using synthetic anomalies.”

2.	Can the authors briefly comment on the use of l1 reconstruction loss instead of l2 ?

We use the l1 reconstruction error as this has shown improved reconstruction quality over the l2 error on the healthy IXI data set in our experiments.

3.	Can the authors precise the window size used for SSIM ?

We calculate the SSIM anomaly score with a Gaussian kernel of size 
k_{ssim} = 9
We added this information to the Manuscript.

4.	Can the authors comment on why it is necessary to register the volumes to a common atlas ?

Registration of volumes to a reference atlas is a preprocessing step commonly applied in UAD. It aims to homogenize the data samples, ensuring they are consistently aligned to a spatial reference frame. While it may not be mandatory, especially in scenarios with ample training data, registration reduces the inter- and intra-data set variability of training and test data sets, which is assumed to simplify the reconstruction and anomaly scoring, particularly when evaluating on multiple different data sets.  

5.	The authors used the sum of the unsupervised anomaly map (x_tilde) and supervised anomaly prediction (y_dot_hat) as anomaly score map. The reviewer would be very curious to see an additional experiment, perhaps in appendix, of any metric, say Dice, plotted against a parameter where the score map used would be BETA * x_tilde + (1-BETA)*y_dot_hat. 

We thank the reviewer for this insightful suggestion. We performed the suggested analysis in the Appendix, Figure 5. Overall, performance and generalization depend on the weighting factor BETA. Setting the beta to 0.5, as done in our experiments, provides a good initial choice and trade-off to generalize across in- and out-domain data. In further work, we plan to investigate more sophisticated mechanisms to combine the anomaly scores and, for instance, incorporate uncertainty estimates of the predictions. 

6.	The reviewer would like the authors to comment the fact that SADM_residual has rhoughly the same performances as SADM_input for the weakly supervised setting. Furthermore, the reviewer believes the sentence "Furthermore, feeding the residual of input and reconstruction to the Unet (SADMresidual )instead of the abnormal input only (SADMinput ) enhances the segmentation performance." has to be nuanced in light of the previous comment.

We acknowledge that performance improvements from SADM_{input} to SADM_{residual} are moderate in the weakly-supervised setting. In contrast, in the self-supervised setting, SADM_{input} substantially impedes the segmentation performance, compared to SADM_{residual}. 
We have revised the discussion: "Furthermore, feeding the residual of input and reconstruction to the Unet (SADMresidual ) instead of the abnormal input only (SADMinput ) can enhance the segmentation performance particularly in the self-supervised setting.”

7.	Although common in UAD for medimage, the reviewer would appreciate if the authors can comment on the fact that the anomalies detected are still quite big and obvious to see (ATLAS and BraTS), and as a perspective refer to more challenging UAD tasks.

Generally, we agree with the reviewer that anomalies in BraTS and ATLAS are rather large and often fairly clear to identify. However, the latter is not equally true for all anomalies. We agree that extending our analysis to further datasets would be interesting.  We added a comment at the end of our manuscript: “Additionally, we intend to include data sets featuring subtler anomalies or different imaging modalities to broaden the evaluation of our approach.”.
Regarding the weaknesses:

While we agree that considering other data sets and body regions would be interesting, we would like to note that the present study is already fairly comprehensive with respect to UAD in brain MRI. Furthermore, we feel that a broader scope would not fit this type of paper. 
Regarding the comparison with the state-of-the-art (SOTA), we believe that we provide a comprehensive assessment by comparing our method to three established approaches based on diffusion models, namely AnoDDPM, pDDPM, and cDDPM.
The reviewer is right that the synthesized anomalies used in DRAEM are not very natural. However, we show that even with these rather unrealistic synthetic anomalies, our proposed framework is able to generalize to real pathologies, which is not the case for the plain self-supervised methods. We expect further performance improvements when using more natural and realistic anomalies in our framework and are curious to pursue this research direction in the future.

Regarding the questions to address in the rebuttal

1.	This sentence is not fully clear: "Instead of learning the removal of synthetic anomalies during training, our model learns to remove any abnormal structure that is not part of the healthy training distribution." How do these learning strategies differ?

We thank the reviewer for pointing this out and will clarify the statement in the following. 
The reconstruction in DRAEM training is enforced to replace the synthetic anomalies by calculating the reconstruction loss between the reconstructed image and the original input image without added anomalies. Therefore, the reconstruction model is directly trained to remove a certain type of anomalies. In contrast, we train our reconstruction model in an unsupervised fashion to reconstruct normal/healthy training data without explicitly enforcing the removal of certain anomalies. 
We adapted the description of DRAEM in section 2 to provide a clearer description in the revised manuscript:
“Note that for the generator network in DRAEM, inpainting of synthetic anomalies is enforced by calculating the reconstruction loss between reconstruction and the anomaly-free input. In contrast, in our approach, the reconstruction model is trained on healthy data in an unsupervised fashion to remove any abnormal structure that is not part of the healthy training distribution.”

2.	Were there any experiments made with 3D instead of slice-wise training, and other imaging modalities?

We conducted experiments to expand the underlying cDDPM reconstruction model to 3D. While observing some performance improvements, training the models in 3D is computationally demanding. Therefore, we focused on slice-wise processing for this study. However, we believe that our method holds promise for application in 3D settings. Also, while it would be interesting to evaluate the presented approach on other datasets and modalities, we had to focus on one scenario for the purpose of this work.

3.	For post-processing, the authors apply median filtering with a kernel size of 5 to the generated anomaly maps. How does this influence the performance regarding tiny/small anomalies?

In general, larger kernel sizes would benefit the detection of larger anomalies while smaller anomalies could be filtered out. However, a kernel size of 5 is a common choice [1,2,3,4] that has not been shown to decrease the performance of smaller anomalies in our experiments. 

4.	How much training time was needed for training stage 1+2? Is it possible to merge both stages into one?

When training SADM_{residual} for synthetic anomalies, the training times for Stage I and II were approximately 5.5 and 2 hours, respectively, on an NVIDIA RTX 3090 GPU. Merging the training stages would be an interesting extension to our approach. We expect that training both stages in an end-to-end fashion is possible but might require a proper weighting of the individual losses. 

Minor comments: 

•	Naming of models could be improved, especially, for 'SADM_{residual}' and 'SADM_{input}', and 'Final Anomaly Map'

Thank you for pointing this out. We changed ‘Final Anomaly Map’ to Anomaly Score (AS). Furthermore, we changed 'SADM_{residual}’ to ‘SADM_{res}’ and 'SADM_{input}’ to ‘SADM’. The same applies to UNET.


[1] Baur et al., “Autoencoders for unsupervised anomaly segmentation in brain mr images: A comparative study.”,2021, Medical Image Analysis 69

[2] Dey et al., “ASC-Net: Adversarial-Based Selective Network for Unsupervised Anomaly Segmentation”, 2021, MICCAI 2021

[3] Akrami et al., “Deep quantile regression for uncertainty estimation in unsupervised and supervised lesion detection.”, 2022, MELBA (1 - IPMI Special Issue)

[4] Kascenas et al., “Denoising Autoencoders for Unsupervised Anomaly Detection in Brain MRI”, 2022, MIDL 2022
1.	Considering that brain images are processed slice by slice, is there a bottleneck when iterating through all slices within a test image volume to generate its segmentation map?

The reviewer is right that sequential slice-wise processing of volumes can slow down the reconstruction process compared to 3D processing. However, we implement the reconstruction in a parallel fashion to reconstruct all slices simultaneously. This reduces the bottleneck, and the reconstruction of a full volume takes approximately 0.5 seconds on an NVIDIA RTX 3090 GPU. 

2.	In Table 1, it appears that the proposed method underperforms in blocks 2 and 3 in the DRAEM test dataset when compared to the baselines. Could you confirm this? Additionally, do you have any intuition for this?

Indeed, regarding synthetic anomalies, our proposed method is outperformed by the fully supervised Unet baselines. This may indicate, that the supervised methods focus on clear on distinct features when detecting anomalies and that the synthetic data show rather distinct differences. Also, the fact that the supervised methods are not superior when applied to the BraTS and ATLAS datasets supports this hypothesis, i.e., the real pathologies may show more subtle patterns.

3.	There is a typo on page 3/Figure 1: “In stage II, the parameters ϕ are fixed,” where ϕ should be θ.

Thank you for pointing this out. We corrected the typo in the revised manuscript.
Dear reviewers, thank you for the detailed reviews and constructive feedback on our work. We are delighted to see that the reviewers appreciated the experimental design, novelty, and contributions our work offers to the field of UAD in brain MRI. Additionally, we are pleased to acknowledge the recognition of our efforts to ensure the reproducibility, clarity and replicability of our research.
We also recognize the identified directions to improve our work. Specifically, questions have been raised regarding the generalizability of our approach to different datasets, pathologies, and imaging modalities. We will thoroughly consider your feedback and answer the questions you raised. Additionally, we provide a revised manuscript that addresses the feedback and includes additional experiments. Changes in the revised manuscript are color-coded in red.
Once again, we express our gratitude for the thorough evaluation of our work and the constructive and detailed suggestions provided. We strongly believe that your feedback allowed us to enrich the quality of our manuscript and provided us the opportunity to steer the direction of our future.