A Patch-based Student-Teacher Pyramid Matching Approach to Anomaly Detection in 3D Magnetic Resonance Imaging

Authors: johannes  schwarz
lena  will
jörg  wellmer
axel  mosig
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 10:16:58

Abstract: Anomaly detection on 3D magnet resonance images (MRI) is of high medical relevance in the context of detecting lesions associated with different diseases. Yet, reliable anomaly detection in MRI images involves major challenges, specifically taking into account information in 3D, and the need to localize relatively small and subtle abnormalities within the context of whole organ MRIs. In this paper, a top-down approach, which uses student-teacher feature pyramid matching (STFPM) for detecting anomalies at image and voxel level, is applied to 3D brain MRI inputs. The combination of a 3D patch based self-supervised pre-training and axial-coronal-sagittal (ACS) convolutions pushes the performance above that of f-AnoGAN (bottom-up). The evaluation is based on a tumor dataset.

Main paper: 
 introduction: 
 In medical image analysis, it is often an attractive and promising approach to view the identification and localization of disease as an anomaly detection problem, i.e., to regard the recognition of disease patterns as the identification of deviations from the norm. While unsupervised or semi-supervised anomaly detection approaches limit the need for extensive annotation, its application in specific medical settings is hampered by several factors. Most current anomaly detection approaches were devised for 2D images and thus do not accommodate the 3D nature of magnetic resonance imaging (MRI). Furthermore, the predominant bottom-up or generative approaches only work well on smaller scaled volumes (Simarro et al., 2020) .We here investigate a novel approach to anomaly detection based on the Student-Teacher model (Bergmann et al., 2019; Wang et al., 2021) . This approach is based on coupling the training process of two convolutional networks: A teacher network, which is trained on normal images as well as images containing anomalies, and a student network, which is trained on normal, anomaly-free images only. The basic idea of a student-teacher network is that when presented with an image containing anomalies, the image representation of the teacher will deviate significantly from the image representation of the student model. An anomaly map is formed by the difference between the layers of the teacher and the student. The conceptual advantage over generative models for detecting anomalies is that the student-teacher is a purely top-down approach and thus allows an explicit definition of a classification loss. As a novel contribution to student-teacher networks, our approach presented here employs triplet margin loss (Vassileios Balntas and Mikolajczyk, 2016) for 3D self-supervised patch learning.A further limiting factor for utilizing the student-teacher approach in medical applications is the reliance on ImageNet-pretrained networks, so that the teacher network cannot be trained with domain-specific knowledge. To address this, our 3D student-teacher feature pyramid matching (STFPM) network uses the top-down structure of the ResNet to connect it to self-supervised patch learning. Here, this technique is applied to the teacher network in the form of patch learning to be able to provide the teacher with a wide range of input data (Danon et al., 2018) .Although not required in our approach, pre-training is still useful to deal with notoriously limited training data in medical applications. To facilitate pre-training in our 3D MRI setting, we employ Axial-Coronal-Sagittal (ACS) convolutions (Yang et al., 2019) , which allow the use of 2D ImageNet weights for 3D convolutions.We evaluate our approach using a data set combining the BraTS tumor data set (Baid and Ghodasara, 2021; Menze et al., 2015; Bakas et al., 2017) with healthy MRI data from the IXI data set (IXI). We show that the use of ImageNet weights and self-supervised patch learning has a major impact on the performance of the 3D STFPM.This combination of the student-teacher approach with patch-based learning and ACS convolutions creates a network that can detect anomalies both at the image level and at the pixel level.To summarize, the main contributions of the paper are: 1. the first approach that extends STFPM to 3D MRI scans; 2. the use of self-supervised patch learning (SSPL) on 3D MRI scans; and 3. the combination of ACS-Convolution and ImageNet pre-trained weights on a medical domain. 
related work: 
 Anomaly detection is often performed via generative networks. This includes GAN-based (Schlegl et al., 2017 (Schlegl et al., , 2019;; Akcay et al., 2018; Donahue et al., 2016) approaches as well as autoencoders (An and Cho, 2015) . In medical imaging on 3D MRI scans, a low resolution is usually selected for GAN-based networks (Schlegl et al., 2019; Siddiquee et al., 2019; Luo et al., 2023) and for autoencoders (Behrendt et al., 2022; Pinaya et al., 2021; Baur et al., 2021) . Slices are often also extracted from the axial, sagittal or coronal and anomaly detection is only operated at slice level (Pinaya et al., 2021; Han et al., 2020) .The student-teacher approach was applied to the MVTec data set and has proven its effectiveness here (Bergmann et al., 2019; Wang et al., 2021) . Self-supervised patch learning Figure 1 : Schematic structure of the 3D student teacher pyramid matching framework, with 3D self-supervised patch learning (p, p + , p -). In contrast to the teacher, the student only knows how healthy tissue is represented in its interlayers. By comparing the intermediate layer outputs from the teacher F t and student F s , a 3D anomaly map A map is generated.comes from the field of metric learning and describes how local image descriptors can be learned. In connection with the student-teacher approach, self-supervised patch learning was already mentioned in Bergmann et al. (2019) . Here, however, it was not applied to medical images and only to 2D data. 
method: 
 In this work, the Student Teacher Model, introduced by Wang et al. (2021) , is used and extended to implicitly learn the feature distribution of healthy and diseased data. Both networks, student and teacher, are based on the same architecture to minimize knowledge loss (Hinton et al., 2015) . The teacher network is pre-trained on healthy and diseased MRI's using self-supervised patch learning. On the other hand, there is the student, which only sees data from healthy patients and is not pre-trained. Pre-training the teacher is done through the following steps: Use Axial-Coronal-Sagittal (ACS) convolution (Yang et al., 2019) to make use of ImageNet weights in 3D convolution and self-supervised image patch learning (Danon et al., 2018) . Then, patch learning allows the teacher to recreate healthy and non-healthy tissue in the feature maps. After the pre-training is complete, the teacher's weights are frozen and only the student's weights are trained. Both the teacher and the student receive an input image, I ∈ R C×X×Y ×Z , where C is the channel dimension, and X, Y and Z represents the 3D image resolution, respectively. Then feature embeddings of the student and the teacher are computed after each ResNet block R i block (with i ∈ {1, 2, 3}). In this way, the student is meant to learn how normal samples are distributed. During testing and evaluation, the difference between feature maps F t and F s are calculated, scaled up, and multiplied which each other and further used as anomaly map A map with the meaning of the higher the difference, the higher the probability of an anomaly.A ResNet is used as the basis of the teacher. All convolutional layers are replaced by ACS convolutional layers, while the pre-trained ImageNet weights are retained.In ACS convolution (Yang et al., 2019) , 2D convolutions are performed in three directions (axial (a), coronal (c), sagittal (s)) of the 3D volume. For this, the 2D kernel is split into three 3D kernels:EQUATIONFrom a 3D input featureI in ∈ R C in ×X×Y ×Z , a 3D output I out ∈ R Cout×X×Y ×Z is createdwith the ACS convolution, which uses the 2D convolutional kernel. C in and C out are the input and output channels, and K denotes the kernel size.Self-Supervised Image Patch Learning After the ImageNet weights have been initialized, the teacher is trained using self-supervised patch learning which, following Bergmann et al. (2019) and Danon et al. (2018) , yields local image descriptors as a result. In this work, we extended the approach from Danon et al. (2018) to 3D input images. For this purpose, anchor boxes p of spatial size (32 × 32 × 32) are randomly cut out for each image I in the teacher training. Then, following a grid, another box p + is cut out in the immediate vicinity (positive box). Negative patches are cut out from another MRI image at a random position. Each of the boxes has a spatial size of (32 × 32 × 32). Anchor boxes that only contain background voxels are discarded.As Bergmann et al. (2019) suggests, in-triplet hard negative mining with anchor swap (Vassileios Balntas and Mikolajczyk, 2016) is used as a loss function that implements an embedding sensitive to the ℓ 2 metric:L teacher = max {0, δ + δ + -δ -} (2)where δ > 0 denotes the margin parameter and in-triplet distances δ + and δ -are defined as:δ + = || T (p) -T (p + )|| 2 (3) δ -= min{|| T (p) -T (p -)|| 2 , || T (p + ) -T (p -)|| 2 } (4)T is the output of the Teacher ResNet.The student's training uses the same architecture as the teacher's to achieve optimal knowledge distillation. This means that all convolution layers are replaced by an ACS convolution.Only, there is no pre-training and the weights are initialized randomly.Training After the teacher has been pre-trained, their weights are frozen and only the student's weights are trained, which is initialized with randomized weights. In each student training step, a batch B of images B = I 1 , I 2 , . . . , I n is fed into the teacher and into the student. Given an input image, I ∈ R C×X×Y ×Z the features F l t (I) and F l s (I) are calculated after each ResNet block l ∈ {1, 2, 3}. To calculate the loss at all positions (x, y, z) in the feature maps, a ℓ 2 distance between ℓ 2 normalized feature vectors is defined and thus the loss over the whole image is calculated via the average at each image position:L l student (I) = 1 X l Y l Z l X l x=1 Y l y=1 Z l z=1 1 2 F l t (I) xyz -F l s (I) xyz 2 ℓ 2 (5)Here X l , Y l and Z l are the spatial resolution of the feature map by ResNet block l. As in Wang et al. (2021) , the features F l t (I) and F l s (I) are respectively normalized to formF l t (I) = F l t (I)) / F l t (I) 2 ℓ 2 and F l s (I) = F l s (I)) / F l s (I) 2 ℓ 2 .Evaluation For an image I that is to be evaluated, the features of the teacher F l t (I) and the student F l s (I) are calculated and then scaled up by trilinear interpolation to the size of the input image I. The upscaled images are each multiplied with one another, resulting in an anomaly map A map . To obtain a detection score s detect for an image I, the maximum value of A map is used: s detect = max (A map ). The entire anomaly map A map is used for pixel level detection. 
experiments: 
 All experiments and their evaluation are performed on the BraTS 2021 (Baid and Ghodasara, 2021; Menze et al., 2015; Bakas et al., 2017) and IXI (IXI) data set. To find the optimal model and show that the combination of ACS convolution, patch learning and pretrained ImageNet weights delivers state-of-the-art performance, the following experiments were performed: Experiment 1: The teacher is initialized with ImageNet with no further teacher training, constituting a fully unsupervised learning setting. Experiment 2: The teacher is not initialized with ImageNet weights, but the self-supervised patch learning is applied. Experiment 3: The teacher is initialized with ImageNet weights, and the selfsupervised patch learning is used in addition. Experiment 4: Same as Experiment 3, but k-means clustering is applied in the evaluation of the A map anomaly map, as suggested by Siddiquee et al. (2019) . Assuming that there is always one contiguous lesion, two clusters can be formed. Namely, one cluster for the healthy tissue and one cluster for the diseased tissue. Since only lesional images are used for this, no detection performance is given here (see table 1 ).For all experiments, ACS convolution is used for the teacher and the student. The experiments aiming to use tumors on the BraTS dataset each use the T2 sequence, since the IXI (IXI) dataset only provides a T2 sequence and no FLAIR sequence. As a reference method for anomaly detection, we use the f-AnoGAN from Schlegl et al. (2019) . The f-AnoGAN is an unsupervised method that detects anomalies both at the image level and at the pixel level. Originally, the f-AnoGAN can only process 2D images. For comparison, all 2D convolutional layers have been replaced by 3D convolutional layers, similar to Simarro et al. (2020) .For comparability, all images are registered on the template MNI-152 (Manera et al., 2020) and skull-stripped with a prefabricated mask to avoid hyperintensities. In addition, histogram standardization (Nyul et al., 2000) and Z-normalization is performed since the MRIs come from different sources (healthy from IXI dataset, diseased from BraTS dataset). The MRI scans are cropped to (156 × 156 × 156) after registration, and then scaled to (224 × 224 × 224) . This is done to eliminate the large black borders around the MRIs that appear just after skull stripping.BraTS and IXI data sets are each split into training (70%), validation (15%), and test (15%) data sets. For the training of the teacher, the same number of MRI scans are taken from the pool of training data of the BraTS and the IXI data set. All IXI images from the training pool are used for the training of the student. All BraTS images from the validation or test pool are used to evaluate the segmentation performance. Equal amounts of IXI and BraTS data from the validation and test pools are used to evaluate the classification performance. The segmentation map of the BraTS data set consists of several regions (no lesion -label 0; non-enhancing tumor core -label 1; the peritumoral edema -label 2; GD-enhancing tumor -label 4). To generate a binary segmentation map, all values greater than 0 are considered a lesion (Baid and Ghodasara, 2021) .To keep the number of parameters as small as possible, a ResNet-18 was selected for the experiments. Further experiments with a ResNet-50 can be found in the appendix (see chapter A). The teacher and the student are each trained for 64 epochs. In the validation, those weights of the student and teacher network were used where the AUROC metric is highest. Such thresholds were then applied to the independent test set. Stochastic Gradient Descent (SGD) with a learning rate of 0.1 is used for the teacher. For the student, the learning rate is 0.5. The batch size is 2 for teachers and students. The training parameters for f-AnoGAN are the same as in Simarro et al. (2020) . 
results: 
 To evaluate performance, we followed common standards and used the area under the reciver-operator curve (AUROC) and Average Precision (AP). In addition, intersection over union and dice-coefficient were calculated to assess segments detected at pixel level. as a backbone and with pre-trained ImageNet weights, using ACS convolution, and 3D self-supervised patch learning beats f-AnoGAN.Metrics Table 1 shows the results compared to the f-AnoGAN. It should be noted that the method with ACS convolution, ImageNet initialization and 3D self-supervised patch learning achieves the best result. Both on the image level and on the pixel level. In addition, one can see that with each additional piece of information that is entered into the training (ImageNet weights, patches for the teacher, and k-Means Clustering), the performance of the independent test data set increases.In the original paper from Wang et al. (2021) , the teacher is only initialized with ImageNet weights. One can clearly see that 3D patch learning provides domain-specific knowledge. In addition, the ImageNet-weight information improves the result again. However, it should also be noted that the evaluation of the intersection over union delivers a poor result. This is because many false positives are produced and no supervised method is used for correction.Parameters The f-AnoGAN with its 3D convolutional layers requires a few more parameters during training. Generator, discriminator and encoder come to 55, 029, 542 parameters. Therefore, the network can only process (64×64×64) voxel images. In contrast, the Student Teacher approach requires only 22,346,752 parameters (11,173,376 each) .Training-Convergence Through early stopping, the optimal AUROC value for detection was achieved after 12 epochs.Through self-supervised patch learning, the teacher learns to replicate the input images exactly in their intermediate layers. The network can thus represent an input image spatially well in the feature space. Pseudo-RGB images can be created by dimension reduction over the multiplied feature embedding layers, to visualize that the Euclidean distances between the individual patches are maintained in the embedding space (Danon et al., 2018) . Looking at the teacher's summed output layers in figure 2 after self-supervised patch learning, one can also see that the network is quite good at tracking healthy and diseased tissue. The pseudo-RGB image, which uses dimension reduction via PCA, also suggests that Euclidean distances are preserved (Danon et al., 2018) . After the first three ResNet blocks, an anomaly map can be created by multiplying the teacher and the student. The figure 3 shows these color-encoded maps belonging to the input image on the left, as well as the intermediate anomaly maps on the different scales.The color-encoding is used to indicate the relative level of anomaly, where blue areas encode low differences and yellow areas encode the highest differences found. The BraTS dataset contains ground truth labels, which encode where abnormal tissue is in the image. When comparing the resulting anomaly map from the network to the ground truth map, one finds that these are similar, suggesting that the network can detect pathological brain formations. 
conclusion: 
 We presented a framework for anomaly detection of 3D MRI scans that uses axial-coronalsagittal convolution to use ImageNet pretrained networks and can simultaneously process a 3D input. With the newly introduced 3D self-supervised patch learning for the teacher, a broad knowledge of healthy and diseased tissue is taught. Together with the student's 3D training, state-of-the-art performance is achieved. Large to medium-sized lesions can be well identified thanks to the top-down approach that can process high-resolution MRI scans.In addition, the framework presented scores with short training times and can therefore be used flexibly.Student-Teacher runtime: 7 minutes per epoch are required to train the teacher. The student is trained in 3 minutes per epoch. For both networks, training is performed over 64 epochs.f-AnoGAN runtime: On the same system, with the same data loader, the f-AnoGAN requires 3 minutes per epoch. The encoder is also trained in 3 minutes per epoch. As suggested by (Simarro et al., 2020) , more than 200 epochs have been trained. 
 
 Appendix:  


Decision: All reviewers found the proposed method to be novel and the results promising. 1 weak accept and 2 borderline accept recommendations from reviewers.

Reviews: review: paper_summary: paper_summary: The paper is interested in, especially small and subtle, anomaly detection in 3D MRI images. It leverages a Student Teacher Feature Matching that operate comparisons at 3 pyramid levels to form an anomaly map. The teacher is trained with both pathological and healthy patients using self-supervised patch learning extended to 3D, and leveraging ACS convolutions to initialize the model with ImageNet weights. The student is trained only with healthy patients. The method is evaluated on brain tumor segmentation using data from the BraTS tumor dataset and IXI dataset for healthy patient data. It compares favorably with respect to state of the art f-anoGAN approach with excellent detection performance and promising segmentation results.
main_review: main_review: The authors answered all my questions, and added details to the paper that improve its readability. I appreciate and thank the authors for their efforts. Despite its clear interest to MIDL attendance, I am not keen on raising my rating to strong accept, which should be reserved for papers that are highly innovative in deep learning, in my opinion.
Please be aware that the paper now slightly exceeds the 8-page limit.
strength_weakness: strength_weakness: - The method is a smart combination of previously published methods, with contributions to adapt them to 3D images. 
- One could expect that more reference methods are included in the experiments, for comparison. In particular, training the proposed model is (very, but still) weakly supervised since the teacher model must be trained with both pathological and healthy patients and the student with only healthy patients. But f-AnoGAN is unsupervised. The performance assessment would be more convincing if a weakly supervised method was included (see e.g. https://arxiv.org/abs/2302.04549 associated with https://github.com/yzhao062/WSAD to take your pick)
- the authors claim their interest in small and subtle anomaly detection, but their method is only applied to brain tumor segmentation. It would have been interesting to have performance assessment on a second clinical question, with smaller anomalies, to support their claim.
- The k-means step, even though it improves the segmentation results, seems like an ad-hoc post-processing step to polish things up. How does it behave if a healthy patient is tested?
- Numerical details are missing concerning data splits for training (section 4.2). How was f-anoGAN trained?
questions: questions: - details on data splits for training models
- the number of false positives seems to be an issue. Could the authors set some more insights on the false positive rate?
- comment on the fairness of the experiments, considering that the proposed model is not purely unsupervised
paper_summary: paper_summary: In this paper, the authors propose to use a student-teacher feature pyramid matching technique (STFPM), previously developped for industrial anomaly detection, and use it, along with two other previoulsy developped techniques (Axial Coronal Sagittal convolutions) and self-supervised patch learning, to detect "anomalies", i.e. brain tumors on the BraTS dataset (public). Healthy training data is the IXI dataset (public). 4 experiments are designed to prove the benefits of some of the techniques added, and a comparison with a state-of-the-art method is done.
main_review: main_review: As described in the comments the reviewer was a little bit disappointed by a small lack of clarity and didactic in the responses but thinks this could still be presented at the conference.
...........
strength_weakness: strength_weakness: 1) The reviewer believes that overall the paper sometimes lacks precise descriptions of the method/evaluation. While these flaws (listed in the categories bellow) are in themselves minor, the reviewer believe that they add up to a point where the quality of the overall paper is hindered.
2) The anomalies detected, i.e. tumors from the BraTS datasets, are quite large and oftentimes very obvious to detect. While this flaw of evaluation is verry common in the current medical imaging UAD literature, the reviewer believes it still should be noted.
3) The novelty is limited, as the 3 methods used and combined here already exist (the reviewer believes that this point in itself would not imply rejection but must still be noted).
questions: questions: - "this patch size turned out to be better." : could the authors precise in what way this smaller patch sized proved better ?
- "Before the patches p, p+ , p− are entered into the teacher one after the other, the patches are scaled to the spatial size of the input images I"  : I did not understand this sentence, are the patches in the end of the same size as the whole image ? If this is the case how is the interpolation done ? Does it make sense to provide a Resnet trained with whole image a patch largely upsampled ?
- F̂sl (I) is here a normalization : what kind of normalization ? Please precise.
- Could the authors precise the scale factor between the three maps of the pyramid ?
- "Experiment 4: Same as Experiment 3, but k-means clustering is applied in the evaluation of the Amap anomaly map, as suggested by Siddiquee et al. (2019)" : the reviewer did not find in the provided reference any mention of k-means. A small sentence precising how is k-means applied and on what would be appreciated.
- "For comparability, all images are registered on the template MNI-152" : is it possible to precise how this registration step is necessary for comparability ? It seems that the developped method could be applied without registration.
- "To generate a binary segmentation map, all values greater than 0 are considered a lesion." : can the authors precise what the labels mean when refering to the BraTS dataset ?
- "For the evaluation, those weights of the student were chosen where the AUROC score is highest" :  can the authors precise this sentence ? AUROC on detection or segmentation ? Is this on the validation set ? If it is on the test set this overestimates the performances.
- The reviewer believes the details on how the detection was done were not provided : the proposed method gives a per-voxel anomaly score map, then how is this score map transformed into a unique score for whole-image detection ?
- The reviewer did not understand why the detection performances were not reported for exp 4
- "Generator, discriminator and encoder come to 55, 029, 542 parameters. Therefore, the network can only process (64 × 64 × 64) voxel images."  : why ? on what laptop/computer configuration ?
- On all the presented visual exemples, from the 3 pyramid maps, the "grosser" systematically missed the tumor, can the authors comment on what this gross-scale map brings to the method ?
- "figure 2 after self-supervised patch learning, one can also see that the network is quite good at tracking healthy and diseased tissue." : can you please precise what the reader is suppose to see ?
- "which uses dimension reduction via PCA, also suggests that Euclidean distances are preserved" : can the authors provide additional details about this experiments ?
- As additional comparison, the reviewer believes the authors could add performances from the literature, i.e. raw AUROC/etc. provided by the papers doing UAD on BraTS, in addition to the re-implementation of f-anogan made by the authors.
- "the framework presented scores with short training times and can therefore be used flexibly." : can the authors provide the training time ? and a comparison with f-anogan ?
paper_summary: paper_summary: The authors propose an anomaly detection framework which uses student-teacher feature pyramid matching for detecting anomalies at image and voxel levels. They integrated self-supervised pre-training with 3D patches and axial-coronal-sagittal convolutions. They evaluate their method on BRATS and IXI.
main_review: main_review: The reviewer thanks the authors for their responsiveness to feedback. With the revisions addressing the reviewer's comments, the manuscript has demonstrated improvement for a 'borderline accept.' For future submissions, the reviewer recommends diversifying evaluations beyond brain datasets and with more benchmarking models to enhance the methods' applicability.
strength_weakness: strength_weakness: - Only f-AnoGan for comparison, more models would have been more convincing
- An ablation study on the ACS convolution would have been interesting, in comparison to 3D convolutions and slice-wise 2D convolutions
- Brain MIR images are less variant, in comparison with, e.g., Chest or Fully-Body MRIs - another dataset with higher structural variance would have been more convincing regarding the generalisability of the proposed method
questions: questions: Were there any preliminary comparisons made with 2D f-AnoGan and a slice-wise version of the proposed method? What would be the expectations of the authors regarding performance and training/inference time?

Were there any experiments made with other data besides Brain MRI datasets?

Comments: comment: The reviewer thanks the authors for the clarifications made.
Thank you for your feedback, we highly appreciated the broad and detailed comments and questions. All major changes have now been highlighted in red. We have made our best possible efforts to explicitly phrase our own choices and reasoning within the character limit for the authors' response. Whenever our choices and reasoning followed previously published arguments, we covered our approach by citing the corresponding reference. In some cases, particularly in the involvement of k-means in the fixed-point GAN, the choices are somewhat hidden in the details of previous work.
The reviewer thanks the authors for the clarifications and answers. The lack of highlighted difference with the former PDF (e.g. modifications in red) and the numbering of questions instead of quoting made the answer a little bit difficult to read. Some parts of the comments were addressed and modified while others remain, which is a little pitiful (e.g. keeping "values greater than 0 are considered a lesion" instead of precising edema, tumor core, etc.). For many comments the authors did not quite explain their reasoning but rather just referred to a reference (e.g. "We followed the practice from [2]"), which the reviewer thinks is a little bit of a shame because it would have been of interest to expose the authors choices and reasoning. Overall many comments still address the questions and modifications have been made accordingly.
- Were there any preliminary comparisons made with 2D f-AnoGan and a slice-wise version of the proposed method? What would be the expectations of the authors regarding performance and training/inference time? 
	- Thank you for the question. In fact, the first experiments were created on 2D slices from the BratS 2021 tumour dataset and the IXI dataset.
	- Details: For each MRI, a 2D slice was cut out of the axial plane where the tumour is largest (compare with segmentation map). For the non-lesional MRIs from the IXI dataset, the registered MRIs were cut out from the same location.
Network architecture was similar to that in ([Wang et al. (2021)](https://arxiv.org/abs/2103.04257)).  The teacher was initialised only with ImageNet weights and pre-trained with self-supervised patch learning in 2D, as in ([Danon et al. (2018)](https://arxiv.org/abs/1807.03130)).
Results (Classification):
ImageNet (AUROC; AP) - 82.0; 85.1
SSPL (AUROC; AP): 82.1; 90.0
	- Evaluation:
Only one slice per MRI was used for training, validation, and testing (where the tumour occupies the largest area according to the segmentation map).
Therefore, the results are good, but not directly comparable to the results in the paper. If you use all slices from an MRI, you can expect significantly worse results because firstly, the spatial information is lost here and secondly, you have to conduct the detection for each slice in the MRI.
Going back to the question of whether this can reduce the training and inference time, we would instead say that it significantly increases it. Extract each slice, send it through the teacher and student, form an anomaly map, and then put all the slices back together to form a large anomaly map.
	- No experiments were conducted for f-AnoGAN on 2D slices.
- Were there any experiments carried out with other data besides Brain MRI datasets? 
	- We attempted [ATLAS](https://fcon_1000.projects.nitrc.org/indi/retro/atlas.html). The detection performance was AUROC 58.2%. No experiments were conducted for segmentation performance because the detection performance does not perform well.
There may be room for further experiments in future work.
- This sentence needs further explanation: "Training-Convergence We have found that after just 12 epochs, the student has trained enough to achieve an optimal AUROC value. With this, good results can be created with little training time." - How? Early stopping or empirically?
	- Thanks for the comment. We have specified the sentence in the paper. Early stopping was used here.
- Q 1:
	- 3D boxes of the size ($32 \times 32 \times 32$) voxels are cut out for the teacher's self-supervised image patch learning. In contrast to the 2D approach from [1], which suggests a patch size of ($16 \times 16$), we found that on 3D MRIs larger boxes lead to better validation metrics. In the paper from [1], the patch size is also introduced as an unexplained hyperparameter.  
- Q 2:
	- Thank you for this important comment. Indeed, this step is unusual, which is why we decided to run all experiments 2 to 4 without upscaling the patches in pretraining (basically, it was only one line of code that needed to be changed).
The results are almost identical. Therefore, we also regenerated all images in the paper for consistency. Thank you for taking a closer look.
- Q 3:
	- This refers to the $ \ell_2 $ normalisation of the features [2]. We made it more precise in the paper.
- Q 4:
	- The feature maps are output according to ResNet blocks 1, 2 and 3 and combined into an anomaly map. The resolutions are (in (c, x, y, z) format: (64, 56, 56, 56), (128, 28, 28, 28), (256, 14, 14, 14), respectively.
- Q 5:
	- While the Fixed-Point-GAN paper by [3] mentions the k-Means step only marginally, the clustering is of quite crucial importance. As thorough inspection of the source code unveils ([Fixed-Point-GAN](https://github.com/mahfuzmohammad/Fixed-Point-GAN/blob/master/solver.py)), k-Means is used for the test results in order to eliminate the many false positives. We followed the same approach for the same purpose.
- Q 6:
	- As the IXI and BraTS data each come from different sources, the established standard pipeline of registration, skullskripping, standardisation and normalisation was used. 
Registration was primarily used to remove the skullcap from the IXI data using the MNI template mask. 
- Q 7:
	- The tumor was divided into different regions in the BraTS dataset: "Annotations comprise the GD-enhancing tumor (ET - label 4), the peritumoral edema (ED - label 2), and the necrotic and non-enhancing tumor core (NCR/NET - label 1)" (see [5]).
- Q 8:
	- We present AUROC for both detection and segmentation (see table 1). We used thresholds obtained from the validation set and applied them to the independent test set (terminology following [Ripley textbook](https://www.stats.ox.ac.uk/~ripley/PRbook/ )), which constitutes the most careful approach to evaluate performance. 
	- The AUROC value for detection is used here for early stopping.
	- We have now specified this in the paper thanks to your comment.
-  Q 9:
	- Image level evaluation: The largest value from the feature embeddings (i.e. anomaly map) is used for this. This value is entered into the evaluation function (AUROC and AP) together with the ground truth. 
	- Pixel level evaluation: Here, the feature embeddings are transferred to the evaluation function (AUROC, DICE, IOU) together with the ground truth map. 
	- We have now specified this in the paper thanks to your comment.
- Q 10:
	- In experiment 4, k-Means was applied based on the knowledge that there is only one contiguous tumor spot per MRI [3]. For the detection performance, whether an MRI is lesional or non-lesional, k-Means has no advantages. 
- Q 11:
	- We have now added the information in the paper (see appendix).
- Q 12:
	- We followed the practice from [2]. This may be of interest to investigate in future work. In our present work, this is not possible due to space constraints. 
- Q 13:
 	- This is an error, it should read Figure 3, not Figure 2. 
- Q 14:
	- Figure 2 basically shows that the feature embeddings also look similar to an MRI input image in the intermediate layers. PCA is used to convert the higher dimensional embeddings into a pseudo RGB image. We have followed Danon et al. [1].  
-  Q 15
	-  The same data set is not always used as a basis. In the example of [4], it is not specified exactly which version of the BraTS data set is used as the basis. The [website (braintumorsegmentation)](http://www.braintumorsegmentation.org/ ) states that the versions differ greatly from one another.
- Q 16:
	- Training student-teacher: 
		- Teacher: 7 minutes per epoch 
		- Student: 3 minutes per epoch 
		- For both networks, the training is carried out over 64 epochs.
	- Training f-AnoGAN: 
		- Generator-Discriminator: 3 minutes per epoch. 
		- Encoder: 3 minutes per epoch. 
		- Just as in the f-AnoGAN paper [6], over 200 epochs were trained.
	- We made it more precise in the paper.

References:
- [1] [Danon et al. (2018)](https://arxiv.org/abs/1807.03130)
- [2] [Wang et al. (2021)](https://arxiv.org/abs/2103.04257)
- [3] [Siddiquee et al (2019)](https://arxiv.org/abs/1908.06965)
- [4] [Luo et al. (2023)](https://www.sciencedirect.com/science/article/pii/S0010482523000756)
- [5] [Baid et al. (2021)](https://arxiv.org/pdf/2107.02314.pdf)
- [6] [Simarro et al. (2020)](https://arxiv.org/abs/2010.04717)
- details on data splits for training models:
	- The T2 sequences of the BraTS and IXI data were each divided into 70\% training data, 15\% & validation data and 15\% test data.
	- We made it more precise in the paper.
- the number of false positives seems to be an issue. Could the authors set some more insights on the false positive rate?
 	- Indeed, as we mention, “many false positives are produced and no supervised method is used for correction”. This comment refers to false activation within MRIs that contain anomalies, rather than false positive identification of non-lesional MRIs. This can also be observed in the f-AnoGAN output ([Simarro et al.](https://arxiv.org/abs/2010.04717)), but is not discussed by the authors. The annotated brain tumors in the datasets may coincide with other anomalies that are not annotated as tumors, which would require detailed discussion. 
- comment on the fairness of the experiments, considering that the proposed model is not purely unsupervised:
	- We highly appreciate raising this question. There is indeed a difference between the prior knowledge required by our approach compared to prior knowledge required in the f-AnoGAN. The f-AnoGAN requires MRIs that are known to be non-lesional for training. Our approach, on the other hand, trains the teacher model on lesional and non-lesional MRIs (without knowing what is what), while only the student requires training on data known to be non-lesional.  
- How was f-anoGAN trained?
	- All parameters were used as suggested by [Simarro et al. (2020)](https://arxiv.org/abs/2010.04717). We have now specified this in the paper.

*Comment:* 
- Thank you for the suggestions. Details about the splits and training details for f-AnoGAN have been added to Section 4.2. Thanks to your comment, it was explicitly pointed out in the paper that k-means is only applied to the lesional MRIs (segmentation task).