Ano-swinMAE: Unsupervised Anomaly Detection in Brain MRI using swin Transformer based Masked Auto Encoder

Authors: kumari  rashmi
ayantika  das
matcha naga gayathri
mohanasankar  sivaprakasam
Venue: MIDL_io_2024_Conference
Published: 2024-02-01 09:01:27

Abstract: The advanced deep learning-based Autoencoding techniques have enabled the introduction of efficient Unsupervised Anomaly Detection (UAD) approaches. Several autoencoder-based approaches have been used to solve UAD tasks. However, most of these approaches do not have any constraints to ensure the removal of pathological features while restoring the healthy regions in the pseudo-healthy image reconstruction. To remove the occurrence of pathological features, we propose to utilize an Autoencoder which deploys a masking strategy to reconstruct images. Additionally, the masked regions need to be meaningfully inpainted to enforce global and local consistency in the generated images which makes transformer-based masked autoencoder a potential approach. Although the transformer models can incorporate global contextual information, they are often computationally expensive and dependent on a large amount of data for training. Hence we propose to employ a Swin transformer-based Masked Autoencoder (MAE) for anomaly detection (Ano-swinMAE) in brain MRI. Our proposed method Ano-swinMAE is trained on a  healthy cohort by masking a certain percentage of information from the input images. While inferring, a pathological image is given to the model and different segments of the brain MRI slice are sequentially masked and their corresponding generation is accumulated to create a map indicating potential locations of pathologies. We have quantitatively and qualitatively validated the performance increment of our method on the following publicly available datasets: BraTS (Glioma), MSLUB (Multiple Sclerosis) and White Matter Hyperintensities (WMH). We have also empirically evaluated the generalisation capability of the method in a cross modality data setup.

Main paper: 
 introduction: 
 The identification and delineation of pathology in brain MRI plays a crucial role in disease diagnosis and prognosis. Unsupervised Anomaly detection (UAD) methods alleviate the task of annotating pathologies at the pixel level. The state-of-the-art deep learning-based UAD methods are Autoencoding models that learn to encode healthy data distribution. The appearance of pathological features in brain MRI scans is typically localized in certain anatomical regions. The Autoencoding networks must ensure changes are introduced in those localized pathological regions while restoring the healthy features. The changes in pathological regions consequently arise due to the fact that Autoencoders are trained on healthy data, and there is a performance degradation in these pathological regions. However, there is no constraint in the Autoencoding models that ensure pathologies will be absent in the pseudo-healthy reconstructed images. To mitigate this, a masking-based Autoencoding strategy is an intuitive approach that can be incorporated. While masking strategies have been adopted for unsupervised anomaly detection (Nguyen et al., 2021) (Iqbal et al., 2023) , most of the approaches do not consider position-aware global context dependencies. These properties are capacitated by Vision transformers (ViT) (Dosovitskiy et al., 2020) , which will ensure that there is an association among distant imaging features of the brain MRI scans (Wang et al., 2023) essential for in-painting the masked regions with meaningful healthy features while removing the pathological ones. However, ViT-based models lack (Xu et al., 2021) (Bietti and Mairal, 2019) inductive biases in modeling local visual structure which cannot be traded with global context since inductive biases for locality information is crucial as we can see in the case of CNN. Additionally, this limitation results in a heavy reliance on large datasets and pre-trained models, which is difficult in scarce data setups like in medical images. To mitigate these challenges, the Swin Transformer (Liu et al., 2021) introduces a shifted windows-based Multi-head self-attention (MSA) for modeling global feature relations. This not only enhances performance but also reduces model complexity. Hence, (i) we propose to utilize a Swin Transformer-based Masked Auto Encoder (Ano-swinMAE) for detecting and localizing pathologies in an unsupervised manner. We have shown the efficacy of our method using publicly available pathology datasets. Additionally, (ii) we have shown how masking helps our models map the pathological distribution closer to a healthy distribution in the latent representational space. (iii) We also validate that our method is generalizable in different datasets and computational efficiency is relatively better. 
related work: 
 Several deep-learning approaches have been investigated in recent research on Unsupervised Anomaly Detection (UAD). Among these, methods incorporating Autoencoders (AE) and Variational Autoencoders (VAE) have proven to be effective during both training and inference (Zhou et al., 2021) , (Baur et al., 2021a) . Nevertheless, a common limitation observed in these approaches is the reconstructed image quality, which tends to be blurry. This blurriness poses a challenge, rendering these methods less effective for UAD tasks, as described in (Baur et al., 2021b) . To overcome this limitation, researchers have worked towards utilizing the image context by adding spatial latent dimension (Baur et al., 2019) , erasing spatial context, (Zimmerer et al., 2018) , making use of 3D information (Bengs et al., 2021) (Behrendt et al., 2022) As an alternative to AE, Generative Adversarial networks have been applied to the problems of UAD task (Schlegl et al., 2019) . However, the unstable training nature of GANs poses challenges, leading to issues such as mode collapse and a lack of anatomical coherence (Baur et al., 2021b) . Meanwhile, the recent work in UAD has utilized DDPM (Denoising Diffusion Probabilistic Model) because it exhibits scalable and stable training properties while producing high-quality, sharp images (Wolleb et al., 2022) (Wyatt et al., 2022) (Sanchez et al., 2022 ) (Pinaya et al., 2022a) . In the DDPM-based approach, there is a tradeoff between preserving crucial information about healthy tissues and efficiently eliminating anomalies due to the inherent noise addition strategy. Recent DDPM works that are used for UAD tasks and deal with this tradeoff are (Behrendt et al., 2023) and (Bercea et al., 2023) . This tradeoff often limits the applicability of the models among diverse pathological data. Additionally, (Iqbal et al., 2023) uses a DDPM-based model which incorporates a mechanism to simulate pathology during the training phase. This strategy involves an approximation of the pathological distribution in the training phase that may not ensure capturing the true distribution.However, the intricate structure of the brain can be captured by learning to model the relationship between individual brain structures, which can be modeled using transformer models. UAD using transformers was done by (Pinaya et al., 2022b) , (Ghorbel et al., 2022) . The transformer-based approaches incorporate masking (Zhou et al., 2023) strategies that ensure the mapping of the pathological brain into a healthy distribution without adding any additional constraints. Employing transformer-based Masked Autoencoders (MAE) (Hu et al., 2023) , (Yu et al., 2022) , (Georgescu, 2023) generate a pseudo-normal counterpart of the normal image. Additionally, (Georgescu, 2023) integrates a pseudo-abnormal module to simulate pathology to train a classifier to discriminate between healthy and unhealthy. Their dependency on the secondary classifier clearly highlights that MAE independently is not suitable for pathology detection. 
problem formulation: 
 Let x ∈ R H×W ×C be a healthy or pathology brain MRI scan with dimensions H × W and C channels. The task of unsupervised detection and localization of pathologies in x can formulated as the reconstruction of the pseudo healthy (x ∈ R H×W ×C ) counterpart of x and extracting the residual information by subtracting x from x. The reconstruction task is achieved by an Autoencoding network, which is trained to map x to x, where x belongs to a healthy cohort input and x is its reconstruction respectively. At the inference time, x belonging to a pathological distribution is given as input to the model and expected to generate x belonging to the healthy distribution.The overview of our proposed Swin Transformer-based MAE (Ano-swinMAE) for anomaly detection is given in Figure 1 . In the training phase, an input to the model (x) is randomly masked at regions within the brain area, which are passed through the encoder to extract latent representations (z ). These representations are further processed through the decoder to generate the reconstructed image, learning to incorporate meaningful information within the masked regions. The masking positions used during training are random but the masking ratio and size of the masks are given by us. The learning objective is a mean square error between the masked region of the original and reconstructed image.The significance of the model lies in the integration of three key components within the autoencoding model. First, a Swin Transformer block (Liu et al., 2021) is introduced within the model along with a patch merging layer which reduces the number of positional information (token). This is quite essential for task which faces data scarcity and requires lesser model complexity. Second, A windowing strategy is incorporated which limits the capability of every patch to be masked. This strategy allows patches within a certain window size to be masked. This masking strategy resembles the pathological occurrence since frequent masking with a small mask size will fail to reconstruct in realistic larger pathological cases. Third, The token of a masked patch is allowed to pass through the model such that the model is aware of the absolute positions of all the patches. This strategy helps in obtaining reconstructed images with better fidelity. We have incorporated all these three strategies and adopted the architectural details of (Dai et al., 2023) to build the encoder-decoder structure of the Ano-swinMAE.In the inference phase, we generate a non-overlapping sliding mask sequentially moving across regions having brain pixels as indicated in the Inference block of Figure 1 . Each masked reconstruction undergoes L1 norm-based computation to derive a residual map compared to its original counterpart. These individual residual maps are then aggregated to form a coarse anomaly map. 
dataset description, pre and post processing, evaluation metrics and implementation details: 
 Dataset Description: We have used four publicly available datasets, IXI (IXI), BraTS21 (Baid et al., 2021) , MSLUB (Lesjak et al., 2018) and WMH dataset (Kuijf et al., 2022) . The IXI dataset(T1, T2) consisting of 580 subjects MRI volume is used as a reference for training our healthy distribution. BraTS21 dataset(T2) consisting of 1251 subjects with MRI volume, and the MSLUB dataset (T2) consisting of 30 subjects with MRI volume, are used for evaluating the performance of our model with different baselines. WMH dataset(Flair) consists of 100 subjects with MRI volume, and the BraTS21 dataset (T1, T2) are used for evaluating the generalizability of the model.Pre and Post Processing: In order to standardize the images over geometric variations, we perform skull-stripping using HD-Bet (Isensee et al., 2019) and rigid body registration with SRI24 (Rohlfing et al., 2010) atlas, effectively resulting the volume to be of 240 × 240 × 155 dimensions. Additionally, to mitigate photometric variations, we perform bias field correction and normalization to the [0,1] intensity range. For post-processing, morphological filters are applied to the coarse anomaly map given by the aggregated residual output of our model to eliminate smaller objects, followed by a connected component-based analysis. This analysis isolates the significant residual components. A Gaussian mixture model-based approach is employed on the original image through Atropos 1 to generate a segmentation mask. Integrating this Atropos segmentation information refines the anomaly map, creating precise segmentation boundaries of the pathologies.Evaluation Metrics: For quantifying the segmentation performance of different models, we have considered the standard definitions of Dice coefficient and Area Under the Precision Recall curve (AUPRC). For analyzing the latent representations of our model, we have projected the high dimensional vectors in a 2d space by using Umap projections (McInnes et al., 2018) . In order to further analyze the spatial relationship of the data, we have performed k-means clustering of the 2d points and extracted the eigenvectors and eigenvalues of the covariance matrix of each cluster. These eigen-components are used to draw an ellipse to represent the spread and orientation of each cluster.Implementation Details and Baselines: Models are implemented in PyTorch 2.0.1 version on an 80GB NVIDIA A100 GPU and CUDA Version: 12.1. For every step mentioned in the training algorithm, models are trained for 400 epochs using the Adam optimizer, with a learning rate of 1e -3 . We have compared our model performance with several existing baselines, such as VAE (Baur et al., 2021b) , f-AnoGAN (Schlegl et al., 2019) , MAE (He et al., 2022) and autoDDPM (Bercea et al., 2023) .We have evaluated on all the baseline methods by adopting the existing implementations. The details of the parameters used in post-processing are mentioned in Appendix A.1. The code for our proposed method will be available at Ano-swinMAE repository 2 .The quantitative evaluations of our method against baselines are summarized in Table 1 . Our model, Ano-swinMAE, shows an increment in performance compared to the other baselines. The effectiveness of different mask sizes in our model varies depending on the dataset: a mask size of 32X32 performs better for MSLUB, whereas a mask size of 64X64 yields superior results in BraTS21. The performance drop of autoDDPM compared to our method could be due to the noising strategy employed by autoDDPM to map pathological distributions to healthy ones. This strategy might not consistently prevent the reconstruction of pathological regions, such as hyper-intense large high-grade gliomas found in the BraTS21 dataset. MAE could not achieve performance increments because of its reliance on a large amount of data, leading to lesser image fidelity compared to Ano-swinMAE. GAN-based approaches often lack one-to-one mappings in healthy brain regions due to modeling challenges like mode collapse. Similarly, VAEs suffer from posterior collapse, resulting in low-fidelity images and an increased occurrence of false positives. Furthermore, Table 1 highlights that our model exhibits better inference time when compared with baselines that have relatively good Dice and AUPRC. The pathology segmentation masks produced by autoDDPM, MAE, and our method for BraTS21 T2 data are illustrated in Figure 2 . We present results across three distinct intensities in pathological regions relative to those in the ventricles and sulci (hyper, medium, and low). Our method and MAE exhibit a consistent trend in performance across these intensity levels, with metrics decreasing in the order of hyper, medium, and low-intensity cases. In contrast, autoDDPM shows a reverse performance trend. Our model demonstrates improved anomaly capture compared to MAE, due to the blurrier reconstruction of MAE that generates more false positives. The autoDDPM model tends to reconstruct traces of pathologies in hyper-intense regions, leading to more false negatives. Additionally, the integration of Atropos-based segmentation information enhances the precision of anomaly boundaries. In Figure 3 , we compare the latent space of masked and unmasked images. From Figure 3 (a) , it is evident that the 2d UMAP projections of the latent representations of Ano-swinMAE tend to separate into two clusters when healthy data slices from IXI and pathological ones from BraTS21 are sent to the model without masking. Each of the ellipses formed from the covariance matrix of each cluster approximately contains projected points from one of the datasets (IXI or BraTS21). Figure 3 (b) indicates that the projections of both datasets tend to collapse within a single cluster. This indicates that since the encoding capability of the model did not explicitly enforce any constraint to map pathological data into a learnt healthy distribution but the masking strategy has enabled the mapping. Consequently, this allows the latent representational space to encode diverse semantics, unlike modeling capabilities like GAN and VAE, which constrain the latent space to follow a prior standard Gaussian distribution. From Table 2 , it is evident that our model Ano-swinMAE shows incremental performance when evaluated on a cross-data setup. When the model is trained on IXI T1 data and evaluated on BraTS21 T2 data, it can perform better than the baseline under a similar setup. Similarly, when IXI T2 is used for training and BraTS21 T2 data for inference, the quantitative metrics are better for our method. In the case of WMH Flair data, our model has slightly inferior results to the baseline. Figure 3 (a), also supports that our model has better Generalization capability since the representational space encodes meaningful semantics tending to form separable clusters for anomaly and healthy data. Ano-swinMAE exhibits better performance when trained on T1 and evaluated on T2 since T2 enhances the pathological appearances in the images and it is easily discriminative. Whereas in the case of autoDDPM if the pathological information is evident, then it is present after the noising process. Further details are discussed in A.3. 
conclusion: 
 In this work, we have proposed Ano-swinMAE, a transformer-based system for unsupervised anomaly detection, which further extends the scope of transformer usage in the field of medical imaging. Our method has outperformed the baselines as well as has shown promising results for generalizable capability. Through latent space analysis, we have observed that masking is quite effective for pseudo-healthy reconstruction of brain MRI. attempts to reconstruct their healthy counterparts. Furthermore, the model demonstrates increased attention when the pathological region is masked. 
 
 Appendix: T2-weighted MRI highlights pathologies like tumors with high water content or those surrounding edema. Hence pathologies mostly become more visually discriminable in T2 modality. Visual results of cross-modality setup are shown in Figure 7 . When, we are assessing the metrics for T2 while being trained on T1, we observe that the performance Images respectively of Ano-swinMAE is better since the model is able to easily differentiate between healthy and pathological content. When the modalities are reversed (inferring on T1 while trained on T2) due to the lesser discriminative appearance of the pathology, the performance of the model drops. There is an analogy between the appearance of the pathology and the performance of the model since it attempts to fill the masked regions with contextual neighborhood information. Whereas in the case of autoDDPM the trend is reversed since having a more evident appearance of pathology (as in T2), the chances to pass the pathological information through the noising process is higher. This degrades the performance scores.  


Decision: All reviewers found the proposed method to be novel, positively evaluated the fact that is based on public data, and the results promising.

Reviews: review: paper_summary: paper_summary: In this paper, the authors developp a method based on masked auto-encoders with swin transformers (Dai et al 2023), for anomaly detection in brain MRI. The method is tested on public datasets Brats, MSLUB and WMH, and also in cross-modality (e.g. train on T1 test on T2). The authors provide additional experiments analysing the latent space with/without masking that strengthen their about the usefulness of masking for learning the healthy data manifold.
main_review: main_review: The comments were all addressed and improved the manuscript overall clarity, which was lacking in the first version. I thus change my grading from borderline to weak accept, as I think this is of interest to present to the conference.
strength_weakness: strength_weakness: 1) The reviewer felt like a lot of claims (pinned bellow) made in the discussion were vague/unprecise. Additional efforts have to be made for clarity of the discussion and to guide the reader towards the main claims of this paper.
questions: questions: - "However, ViT-based models lack (Bietti and Mairal,2019) inductive biases which cannot trade with global context since inductive biases are crucial for locality information." : can the authors precise what inductive bias they are thinking about ?
- "and image reconstruction tradeoffs used for UAD tasks are (Behrendt et al., 2023) and (Bercea et al., 2023). This tradeoff often limits the applicability of the models among diverse pathological data" : can the authors precise what tradeoff they are thinking about ?
- The method seem to be very similar (SwinMAE) to the method of Dai et al 2023, can the authors list explicitely the difference with this method ?
- For the post-processing (morpho maths, connected components, etc.) could the authors provide the exact operations in appendix for reproducibility ?
- The atropos framework is used for coarse segmentation, which is then said to "refine the anomaly map", can the authors give visual exemples or precise what type of refining is done with this segmentation ? Moreover, on Figure 1 the structures (mixture components) are ranked in such a way that 1 component has higher intensity than another, etc. How is this ordering done ?
- Can the authors precise if the performances for the other competiting methods are taken from the literature or are the methods reimplemented ?
- The AU PR is used as a metric, the baseline for this metric is not 0.5 as for the AU ROC, can the authors provide the baseline (obtained with a random classifier) for the datasets used ?
- Is the inference time per volume or for the whole dataset ? Or else ?
- The reviewer found that the whole block beginning with "MAE could not achieve performance" and ending with "resulting in low-fidelity images and an increased occurrence of false positives." was very unclear. Can the authors detail each sentence, precise what they mean exactly ? The interpretation seems interesting but the reviewer feels like as is it is not understandable.
- "Notably, the integration of superpixel information enhances the precision of anomaly boundaries." : can the authors precise what they meant ? The term "superpixel" first appears in this sentence.
- "Each of the ellipses formed from the covariance matrix of each cluster mostly contains projected points from one of the datasets (IXI or BraTS21)" : I would change "mostly" to "rhoughly", as one of the ellipse containt both purple and yellow dots.
- Table 2 : with which patch size are the results obtained ?
- Can the authors comment on the fact when training on IXI and testing on brats for cross-modality (table 2), the results are very different if T1 -> T2 or T2 -> T1 ?
- "Consequently, this allows the latent representational space to encode diverse semantics, unlike modelling capabilities like GAN and VAE, which restrict the latent space." : again, can the authors precise what they meant ? What is restricted in GAN and VAE that is not in Transformers ?
- The main text, and the legend of the Figure 4 and 5 have to explain what is done in these figures. As is it is not understable, although it looks interesting.
paper_summary: paper_summary: Authors propose an unsupervised anomaly detection approach to detect pathological regions in brain MRI data. The approach is based a Swin-tranformer-based Masked Autoencoder model trained to reconstruct healthy images. At inference, pathological regions are detected by subtracting the reconstructed pseudo-healthy images and the input pathological image using a sliding window algorithm. The method is evaluated on three publicly available datasets, namely BraTS, MSLUB, and WMH and compared to other generative and autoencoding approaches.
main_review: main_review: The rebuttal provided clarifications on my concerns regarding the pre-/post-processing and implementation details of the methodology, and are quite heavy. 
However, the strengths/weaknesses balance remains similar to me. Hence the choice of slightly increasing my rating to borderline accept.
strength_weakness: strength_weakness: - Similar related work, including masking strategies for anomaly detection, with diffusion models for medical images [A], with transformer models in industrial images [B, C], and for medical images, especially brain MRI data [D], are missing in reference to prior work and should be discussed by authors, how their approach compares to these;
- Methodology (and pre- and post-processing) misses some precise implementation details, which hamper reproducibility, as well as code is not available ("will be available soon").

[A] Iqbal et al. Unsupervised Anomaly Detection in Medical Images Using Masked Diffusion Model. Workshop MLMI. 2023
[B] Hu et al. Features Masked Auto-Encoder-Based Anomaly Detection in Process Industry. DDCLS. 2023.
[C] Yu et al. Image Anomaly Detection and Localization Using Masked Autoencoder. ICONIP. 2023.
[D] Georgesc. Masked Autoencoders for Unsupervised Anomaly Detection in Medical Images. Procedia Computer Science. 2023
questions: questions: - Authors should better incorporate and discuss references to prior work using masking strategies for anomaly detection. This includes specifying the contributions of the proposed approach;
- The methodology description should be more precise, especially regarding implementation details of the pre- and post-processing, the sliding window, and the use of the segmentation model; code availability would highly help;
- Answer major comments.
paper_summary: paper_summary: The authors address the difficult task of unsupervised anomaly detection in human brain MRI. To tackle this issue they propose a masking auto-encoder approach based on a Swin transformer architecture to contraint the model to remove pathological features when reconstructing a pseudo-healthy version of the patients' MRI, while not deteriorating the healthy regions. The model was trained on a dataset with healthy MRI only, and then the inference and evaluation was performed on three publicly available MRI datasets from patient with Glioma (BraTS), Multiple Sclerosis (MSLUB) and White Matter Hyperintensities (WMH).
main_review: main_review: I appreciate the clarity improvements brought by the edits the authors did on the paper. The post-processing was an obvious topic of discussion for this new version of the paper. Despite a valid answer to my question about this post-processing "imbalance" to present a fair comparison of the compared methods, the authors do not demonstrate that it is indeed their mainly claimed contribution ("utilize a Swin Transformer-based Masked Auto Encoder (AnoswinMAE) for detecting and localizing pathologies in an unsupervised manner") and not the post-processing that brings the positive performance gap for the proposed method... comparing with/without the post-processing would have been a very convincing answer. Grading unchanged.

Figure 5 and Figure 6 : Row(ii) and (iii) a=> Row (ii) and (iv)
strength_weakness: strength_weakness: The meta parameter defining the mask shape is critical as it impacts the results and inference time. Additional values of masks size (16x16, 128x128 ?) and a discussion about the trade-off and adaptation to the dataset of use would be an additional valuable contribution.

The section 4.2.3 about generalization across modality shows promising results, but it misses the illustration of the latent space projections and some segmentation maps outputs as in Fig 2. They could be added in Appendix for instance.

The (potentially true) hypothesis on why the other methods yield lower results than the proposed ones are not supported by experiments, please adopt a less confident tone when taking the discussion on these hypothesis (can  => could, etc ...).

Description of Fig 4 and Fig 5 are not precise enough: please add letters on the image samples and attention maps and link their description in the legend accordingly. Guide the reader to better understand what the attention maps is supposed to look at in each case.   
Fig 4 : why not displaying the results of the 64x64 masking as the results are better ?

The preliminary results obtained on cross-modality are promising but from the T2 (learn) to T1 (infer) the domain shift seems to be better managed by autoDDPM, can you comment on this ?

The post-processing does not seem to be applied to the outputs of autoDDPM (Fig.2 (c)), same for the Atropos segmentation mask integration (post processing described on page 5). Please also apply the post-processing steps to the autoDDPM (and other concurrent methods) outputs and recompute the metrics accordingly.
questions: questions: The question to address were grouped in the Weaknesses section alongside with the points to treat.

Can you comment about the inference time that is longer for Ano-swinMAE (32x32) compared to its (64x64) version (Table. 1, page 6) ?

Comments: title: Response to rebuttal
Addressing The Detailed Comments
Addressing The Rebuttal Questions Set2
Addressing the Rebuttal Questions Set1
Addressing The Rebuttal Questions
Addressing  The Rebuttal Questions
comment: The reviewer thanks the authors for the clarifications and responses made to the concerns.
Thank you for your detailed answer and the modifications made to the manuscript. 
Clarifications were made, especially on the pre-/post-processing parts.
We have addressed the detailed comments of the reviewer as follows:

[1] "have proven to be effective during both training and inference."..
> We have added two references (1) “Autoencoders for unsupervised anomaly segmentation in brain MR images: A comparative study” and (2) “VAE-based Deep SVDD for anomaly detection” in the manuscript, which are consolidated review articles on Autoencoder-based unsupervised anomaly detection.

[2] Figure 1: "psedo" -> "pseudo"
> We have corrected the typo in Figure 1.

[3] "x to x̂, both of which belong to a healthy cohort."..
> We have addressed this by referring to  x̂  as the reconstructed image instead of claiming that it belongs to the healthy cohort.

[4] "The token of a masked patch is allowed to pass through the model such the model" --> such as ?
> Here we have added a connective word “such that” instead of “such”.

[5] "Area Under the Precision Accuracy curve (AUPRC)" -> Recall instead of Precision
> Thanks for highlighting this typo, we have changed Accuracy to Recall.

[6] "The code for our proposed method will be available soon." …
> We have provided the link for traceability https://github.com/rashmi05pathak/Ano-swinMAE

[7] "The pathology segmentation masks produced by autoDDPM, MAE, and our method for BraTS21 T2 data are illustrated in Figure 5." I think the authors meant Figure 2. 
 > The overlapping of Figure numbers(2 & 5) is due to the same label assignment (\label{} in latex) to the figures. We have changed the label assignment of the figures.

[8] "The autoDDPM model tends to generate pathologies" --> detection, or other wording
> It is a valid concern. We have changed the word “generate pathologies” to “reconstruct traces of pathologies”.

[9] Figure 2: The reviewer is unsure but thinks it would be more intuitive to have the TP in green. 
> We understand the concern and have changed TP to green, FP to blue and FN to red. 

[10] "In the case of WMH Flair data, our model has comparative results to the baseline." : I would add "and slightly worse AUPRC" 
> We understand the concern that comparative doesn't exactly quantify the performance. We have added “slightly inferior” instead of comparative.

[11] "tending to form differentiable clusters": I would not use the term "differentiable"..
> We have replaced the term “differentiable” with “separable”.

[12] "which further extends the scope of transformers in the field of medical imaging": I would add "transformer usage"
> We have replaced “the scope of transformers” with "transformer usage".
[10] The term "superpixel" first appears in this sentence.
> Yes, indeed the term appears for the first time. We meant to denote that the accumulation of similar intensity with Atropos segmentation enhances the precision of anomaly boundaries. We have rephrased “Notably, the integration of superpixel information enhances the precision of anomaly boundaries.” with “Additionally, the integration of Atropos-based segmentation information enhances the precision of anomaly boundaries” in the paper.

[11] I would change "mostly" to "rhoughly", as one of the ellipse contain both purple and yellow dots.
> Yes we understand the concern, We have addressed this by changing “mostly” to “approximately” in the paper.

[12] Table 2 : with which patch size are the results obtained ?
> It is a valid point and needs to be mentioned in the description. We have added it in the description of Table 2. The patch size we have performed generalization with is 64 X 64.

[13] Can the authors comment on the fact when training on IXI and testing on brats for cross-modality …
> It is a valid concern and needs to be highlighted. T2-weighted MRI highlights pathologies like tumors with high water content or those surrounding edema. Hence pathologies mostly become more visually discriminatable in T2 modality. When, we are assessing the metrics for T2 while being trained on T1, we observe that the performance of Ano-swinMAE is better since the model is able to easily differentiate between healthy and pathological content. When the modalities are reversed (inferring on T1 while trained on T2) due to the lesser discriminative appearance of the pathology, the performance of the model drops. There is an analogy between the appearance of the pathology and the performance of the model since it attempts to fill the masked regions with contextual neighborhood information. We have supported our claim by adding an image in the Appendix (Figure 7). In the case of autoDDPM the trend is reversed since having a more evident appearance of pathology (as in T2), the chances to pass the pathological information through the noising process is higher. This degrades the performance scores.
We have added this reasoning in SubSection 4.2.3 as “ Ano-swinMAE exhibits better performance when trained on T1 and evaluated on T2 since T2 enhances the pathological appearances in the images and it is easily discriminatable. Whereas in the case of autoDDPM if the pathological information is evident then is present after the noising process.”

[14]  What is restricted in GAN and VAE that is not in Transformers ?
> We acknowledge the necessity of mentioning the details about restriction in latent space. We have rephrased the previous content with the following and added it to the manuscript --- Consequently, this allows the latent representational space to encode diverse semantics, unlike modeling capabilities like GAN and VAE, which constrain the latent space to follow a prior standard Gaussian distribution.

[15] The main text, and the legend of the Figure 4 and 5…
> We thank the reviewer for going through the appendix. We have added a section in the Appendix (A.2.) which describes the information presented in the Figures.
First of all, we thank the reviewer for their insightful and precise comments which will definitely elevate the quality of the paper. We have addressed the Detailed Comments and highlighted them in the manuscript. The details of each of the comments are discussed below:


[1] "However, ViT-based models lack (Bietti and Mairal,2019) inductive biases…
> We acknowledge that precisely mentioning “inductive bias” is necessary. Here inductive bias refers to the modeling of neighborhood/ locality structures which is an implicit property of a Convolutional Neural Network (CNN). The convolution operations in CNNs extract local features from the neighbor pixels within the receptive field determined by the kernel size. Inductive Bias follows the intuition that local pixels are more likely to be correlated in images. We have cited an additional reference in the paper and rephrased  “However, ViT-based models lack (Bietti and Mairal, 2019) inductive biases which cannot trade with global context since inductive biases are crucial for locality information.”  in the following manner -- “However, ViT-based models lack (Xu et al., 2021)(Bietti and Mairal, 2019)inductive biases in modeling local visual structure which cannot be traded with global context since inductive biases for locality information is crucial as we can see in the case of CNN”.


[2] "and image reconstruction tradeoffs used for UAD tasks …
> We understand that clarity is required in this phrase. To address this, we have added this line in the paper “In the DDPM-based approach, there is a tradeoff between preserving crucial information about healthy tissues and efficiently eliminating anomalies due to the inherent noise addition strategy”. With the increase in noising time steps, the chances of removal of pathological information increases while there is a compensation of the amount of healthy region that will be reconstructed back.

[3] The method seem to be very similar (SwinMAE)...
> It is a valid concern. Our architectural proposition has similarities to Dai et al 2023, and their proposition is to derive a downstream task out of the pre-trained swinMAE network. whereas we propose to use the model's implicit ability to fill masked patches with information from the learnt distribution. Consequently, we propose tackling the anomaly-to-pseudo-healthy generation task by incorporating this inherent capability of the model.


[4] For the post-processing ..
> We completely understand the necessity of adding these details. We have added the detailed post-processing steps and the sliding window method in the Appendix section (Appendix A.1) of the paper. 

[5] The atropos framework is used for coarse segmentation ...
> We acknowledge the importance of highlighting the significance of adding Atropos framework. We have included the images in the Appendix section. The Atropos framework is used to refine the boundaries of pathologies by incorporating the segmented information from the original MRI slice. In Figure 1, we have not adopted any ranking based on intensities, whereas in Figure 2, the ranking is done based on the visual intensity appearance of the pathological regions in T2 imaging modality compared to other healthy bright regions (CSF, sulci). 

[6] Can the authors precise if the performances for the other competiting methods …
> We understand the significance of highlighting this. We have reimplemented all other baseline methods.

[7] The AU PR is used as a metric, the baseline for this metric is not 0.5…
> Yes the AUPRC metric does not have a fixed baseline but rather depends on the fraction of positives present within a dataset. We have used a random classifier for BraTS21 dataset and inferred the following: We have 3-4% of the pixels/slice (average) with pathologies in the entire dataset, evaluating with a random classifier the AUPRC scores range from 0.02 to 0.2. It is to be noted that the randomness of the classifier introduces a range of scores.

[8] Is the inference time per volume or for the whole dataset ? Or else ?
> It is a valid concern, we have added (per MRI slice) in Table 1.

[9] The reviewer found that the whole block beginning with "MAE could not achieve performance" ..
> We acknowledge that the explanation should be more clear. We have replaced “MAE could not achieve performance increments due to its dependence on a large volume of data for convergence, generating images with relatively lower fidelity.” with “ MAE could not achieve performance increments because of its reliance on a large amount of data, leading to lesser image fidelity compared to Ano-swinMAE”. The MAE model is dependent on a large amount of data, when we train it with lesser data (as we are using medical images) the model is not able to reconstruct good-quality images.
We sincerely thank the reviewer for their insightful comments and for acknowledging the strength of the manuscript. We have incorporated suggested changes in the manuscript and highlighted the changes. We addressed the questions,  Major and Minor comments as follows:

[1] Authors should better incorporate and discuss references to prior work …

> We thank the reviewer for pointing to such relevant references. We have added the details of the cited papers in the Related Work section. The primary difference between the cited references and our paper is described below:
(i) Addressing paper [A], in this work a diffusion model is used, and during the training phase, a mechanism to simulate pathology is introduced. The simulation is done by masking the original image and adding a frequency-altered version of the original image in the masked images. Their notion of introducing masks is to simulate pathologies unlike ours where we intend to mask the pathological image during inference.
(ii) Addressing paper [B][C][D], these works employ transformer-based Masked Autoencoders (MAE) to generate a pseudo-normal counterpart of the normal image. Additionally, [D] integrates a pseudo-abnormal module to simulate pathology to train a classifier to discriminate between healthy and unhealthy. Their dependency on the secondary classifier clearly highlights that MAE independently is not suitable for pathology detection.

[2] The methodology description should be more precise…
> We acknowledge the necessity of adding these precise details. For pre-processing, we have used (i) skull-stripping (ii) Registration and (iii) Linear Normalisation (0-1). We have added the detailed post-processing steps and the sliding window method in the Appendix section (Appendix A.1) of the paper. We intend to make the code available upon acceptance. We have provided an empty repo link for traceability  (https://github.com/rashmi05pathak/Ano-swinMAE). 

[3] Not sure the name of the proposed method is a correct keyword;
> It is a valid concern. We  Ano-swinMAE refers to the task of dealing with Anomaly detection using a swin transformer-based Masked Autoencoder.


[4]Section 3: The described methodology is for 2D images, while brain MRI…
> It is a valid concern. We are working with 2d slices of the 3d brain mri.

[5] Section 4: "For post-processing, morphological filters are applied" ..
> We thank the reviewer for highlighting this. We have added the detailed post-processing steps and the sliding window method in the Appendix section (Appendix A.1) of the paper. We have done post-processing based on morphological operations and connected component analysis which is generally adopted for Unsupervised brain Anomaly Detection. The additional component in our post-processing step is adding information from Atropos segmentation which only contributes to refining the boundary information. We have included a Figure in the Appendix that shows the boundary refinement due to the addition of Atropos segmentation. We have evaluated all the baseline methods by adopting the existing implementations available. 

[6] Section 4: It's not clear how the Atropos segmentation …
> It is a valid concern. We have added the detailed post-processing steps and the sliding window method in the Appendix section (Appendix A.1) of the paper. 

[7] Appendix figures are not discussed…
> We thank the reviewer for going through the appendix. We have added a section in the Appendix (A.2.) which describes the information presented in the Figures.

[8] Section 4: "normalization to the [0,1] intensity range." linearly?
> Yes we have performed a linear intensity normalization.
We sincerely thank the reviewer for acknowledging the strengths of our manuscript and for highlighting points that can improve the understanding of our manuscript. We have incorporated suggested changes in the manuscript and highlighted the changes. We have addressed the weaknesses and questions as follows:


[1] The meta-parameter defining the mask shape is critical …
> We acknowledge the importance of tuning the hyperparameter and analyzing the results at more number of mask sizes. We have performed experiments on MSLUB and BraTS21 dataset with mask size 128X128. The Dice score (%) obtained for MSLUB is 17.42%, and for BraTS21 it is 40.25%. The AUPRC score (%) obtained for MSLUB is 12.32%, and for BraTS21 it is 27.91% Hence comparing with mask size 64X64 we have a performance drop in both cases. The increase in mask size from 64 to 128 covers most of the regions of the brain which hinders the reconstruction capability. Additionally, with mask size 16X16 the Dice and AUPRC for MSLUB is 18.95% and 12.44% respectively, when comparing it with the results of 32X32, there is a drop in performance. 
 Hence the optimal mask size for MSLUB is 32X32, and for BraTS21 is 64X64, which shows that this hyperparameter is dependent on the size of the pathology we are dealing with.  However, the computational complexity increases with the decrease in mask size.


[2] The section 4.2.3 about generalization across modality shows promising results…
>  It would definitely be an important addition. We have added a figure (Figure 7) in the Appendix which shows the results in the cross-modality setup.

[3] The (potentially true) hypothesis on why the other methods yield..
> We completely agree with the comment of the reviewer. We have changed ‘can’ with ‘could’ and ‘may with ‘might’ in the Quantitative And Qualitative Analysis section where we stated our hypothesis.

[4] Description of Fig 4 and Fig 5 are not precise enough…
> Figures 4 and 5 have had their captions revised to enhance clarity. Additionally, a new section "Visualization of Attention Maps" has been included in the Appendix, providing detailed descriptions of the attention maps.

[5] Fig 4 : why not displaying the results of the 64x64 masking...
> We have added a Figure (Figure 6) in the Appendix displaying the results of 64x64 masking.

[6] The preliminary results obtained on cross-modality are promising...
> It is a valid concern and needs to be highlighted. T2-weighted MRI highlights pathologies like tumors with high water content or those surrounding edema. Hence pathologies mostly become more visually discriminatable in T2 modality. When, we are assessing the metrics for T2 while being trained on T1, we observe that the performance of Ano-swinMAE is better since the model is able to easily differentiate between healthy and pathological content. When the modalities are reversed (inferring on T1 while trained on T2) due to the lesser discriminative appearance of the pathology, the performance of the model drops. There is an analogy between the appearance of the pathology and the performance of the model since it attempts to fill the masked regions with contextual neighborhood information. We have supported our claim by adding an image in the Appendix (Figure 7). In the case of autoDDPM the trend is reversed since having a more evident appearance of pathology (as in T2), the chances to pass the pathological information through the noising process is higher. This degrades the performance scores.
We have added this reasoning in SubSection 4.2.3 as “ Ano-swinMAE exhibits better performance when trained on T1 and evaluated on T2 since T2 enhances the pathological appearances in the images and it is easily discriminative. Whereas in the case of autoDDPM if the pathological information is evident, then it is present after the noising process.”

[7] The post-processing does not seem to be applied ...
> We thank the reviewer for highlighting this. We have evaluated all the baseline methods by adopting the existing implementations available in the official repositories, except for MAE which was adopted from a paper demonstrating its applicability in the classification of natural images.

[8] Can you comment about the inference time that is longer...
> We acknowledge the significance of reasoning out the difference in the inference time. The increase in inference time for 32x32 mask size is due to the increase in the number of masked images that has to be processed.

[9] page 2, last paragraph, "DDPM" : undefined acronym...
> It is a valid concern. We have added the definition of the acronym in the Related Work section.


[10] Double check the paper spelling...
> We definitely need to address these typos. We have replaced “Psedo” with “Pseudo”. We changed the title of section 4.2.2 as Analysing The Effect Of Masking in The Latent Space. Also added a line “ In Figure 3, we compare the latent space of masked and unmasked images.” in the description.