Scale-Aware Contrastive Reverse Distillation for Unsupervised Medical Anomaly Detection
Authors: Chunlei Li
Yilei Shi
Jingliang Hu
Xiao Xiang Zhu
Lichao Mou
Venue: ICLR 2025 Poster (2025)
Keywords: unsupervised anomaly detection
medical images
contrastive reverse distillation
student-teacher
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Unsupervised anomaly detection using deep learning has garnered significant research attention due to its broad applicability, particularly in medical imaging where labeled anomalous data are scarce. While earlier approaches leverage generative models like autoencoders and generative adversarial networks (GANs), they often fall short due to overgeneralization. Recent methods explore various strategies, including memory banks, normalizing flows, self-supervised learning, and knowledge distillation, to enhance discrimination. Among these, knowledge distillation, particularly reverse distillation, has shown promise. Following this paradigm, we propose a novel scale-aware contrastive reverse distillation model that addresses two key limitations of existing reverse distillation methods: insufficient feature discriminability and inability to handle anomaly scale variations. Specifically, we introduce a contrastive student-teacher learning approach to derive more discriminative representations by generating and exploring out-of-normal distributions. Further, we design a scale adaptation mechanism to softly weight contrastive distillation losses at different scales to account for the scale variation issue. Extensive experiments on benchmark datasets demonstrate state-of-the-art performance, validating the efficacy of the proposed method. The code will be made publicly available.

Published as a conference paper at ICLR 2025
SCALE-AWARE CONTRASTIVE REVERSE DISTILLA- TION FOR UNSUPERVISED MEDICAL ANOMALY DE- TECTION
Chunlei Li
Yilei Shi
MedAI Technology (Wuxi) Co. Ltd. chunlei. wiimedimmgingaaaai com
MedAI Technology (Wuxi) Co. Ltd. yilei. shi tandimmmimigga..... com
Jingliang Hu
Xiao Xiang Zhu
MedAI Technology (Wuxi) Co. Ltd. jingliang
Technical University of Munich xiaoxiang. zhuê thuetun um. de
com
Lichao Mou
MedAI Technology (Wuxi) Co. Ltd. 1i Lichao. mou/@medimagingaii com.
ABSTRACT
Unsupervised anomaly detection using deep learning has garnered significant re- search attention due to its broad applicability, particularly in medical imaging where labeled anomalous data are scarce. While earlier approaches leverage gen- erative models like autoencoders and generative adversarial networks (GANs), they often fall short due to overgeneralization. Recent methods explore various strategies, including memory banks, normalizing flows, self supe-supereieed learning. and knowledge distillation, to enhance discrimination. Among these, knowledge distillation, particularly reverse distillation, has shown promise. Following this paradigm, we propose a novel scale-aware contrastive reverse distillation model that addresses two key limitations of existing reverse distillation methods: in- sufficient feature discriminability and inability to handle anomaly scale varia- tions. Specifically, introduce contrastive student-teacher learning approach to derive more discriminative representations by generating and exploring out-of- normal distributions. Further, we design scale adaptation mechanism to softly weight contrastive distillation losses at different scales to account for the scale variation issue. Extensive experiments on benchmark datasets demonstrate state- -h-the-art performance. validating the efficacy of the proposed method. Code is available url https::/ thub. MMod/ Tech SCRD4AD
INTRODUCTION
The automatic detection of anomalies medical images crucial yet challenging task. Find- ing abnormalities early through screening enables timely intervention and improves patient out- comes (Cai al., (202)). However, designing robust algorithms for this task is difficult due the variability in anomalous anatomy across patients. Moreover, obtaining annotated data with verified anomalous samples is often prohibitively expensive and time-consuming (Cai al. 2023 Schlegl et al. 2019). Consequently, there is pressing need for unsupervised anomaly detection methods capable recognizing anomalies without relying on labeled abnormal training data.
Early efforts utilize generative models such autoencoders and generative adversarial networks (GANs) (Schlegl et al., 2019 Jiang et al. 2019 Han et al., 2021a [Shvetsova et al., (20T)). These models trained to leam feature representations exclusively from normal images. Abnormalities
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
can then be detected by determining if test images lie outside the manifold of the learned represen- tations, or by comparing original and generated images in pixel space. The underlying hypothesis is that a generative model trained on normal samples can accurately reconstruct anomaly- regions well but struggles with anomalous ones. Nevertheless, this does not always hold, and generative models often overgeneralize, that is, they tend to generalize too well, thereby risking the reconstruc- tion of abnormal regions (Gong et al.) 2019).
To address this issue, some approaches introduce memory banks that store representative normal patterns to help control model generalization (Gong et al. 2019). At test time, images are directly compared to the memory banks identify anomalies. In addition, several studies use normalizing flows (Rudolph et al. [02T] Yu et al.) [022]] Guudovskyy et al., 2022). A normalizing flow models a target distribution as an invertible transformation of a base distribution (e.g., Gaussian) in latent space. In the context of anomaly detection, the flow is trained to maximize the likelihood of normal patterns, and the likelihood cannot be simultaneously increased for all images. By doing so, it as- signs positive density only to normal samples and, in particular, does not generalize to anomalous ones. Concurrently, self-supervised learning aJing & Tian (02T)) has catalyzed the development of unsupervised anomaly detection algorithms (Li et al. [0.21 (Sonn et al.) 2021 Schlüter al., 2022). Existing self-supervised learning-basee methods typically follow one of two paradigms: one-stage or two-stage approaches. In the one-stage approach, model is trained to detect artificially synthe- sized anomalies and then directly applied to detect real abnormalities. The two-stage approach, on the other hand, first learns sulf-super representations through a pretext task on normal data and subsequently constructs a one-class classifier based on the learned representations. However, these methods still exhibit limited discriminative capabilities in real-world medical anomaly detection and incur heavy computational overheads.
Knowledge distillation from pre-trained models presents another promising approach for unsuper- vised anomaly detection (Salehi et al. (202T). This methodology typically employs teacher-student paradigm, where the teacher an encoder network pre-trained on a large-scale dataset (e.g., Ima- geNet), and the student network has similar or identical architecture. The key insight is that the stu- dent is exposed only to anomaly-free images during knowledge distillation, lcading to discrepancies between features of the teacher and student networks when encountering anomalies during infer- ence. Knowledge distillation- based approaches reconstruct features of pre-trained encoders rather than raw pixels, as features provide more informative representations and yield superior results. To further enhance the discriminative capability of the teacher-student framework, various strategies are explored. For example. [Bergmann al. (2020) ensemble several student networks trained nor- mal images at different scales, while Salehi et al. (202T) and Wang et al. (202T) leverage multi-level feature alignment. Deng & Li (2022) propose an interesting reverse distillation model that adopts a heterogencous teacher-student framework, comprising teacher encoder and student decoder. This method distills knowledge from the pre-trained teacher network into the student network in a reverse direction, achieving better performance.
In this paper, we approach the problem of unsupervised anomaly detection through the lens of re- verse distillation. We identify two key limitations of the reverse distillation model. First, knowledge distillation alone is insufficient to provide discriminative representations to the student network. Second, we observe that anomalies vary in size, posing a challenge to scale-equalizing knowledge distillation models. To address these issues, we propose a scale-aware contrastive reverse distillation model. Our contributions are threefold:
We propose a contrastive student-teacher learning method within the reverse distillation paradigm, aimed deriving discriminative representations by generating and exploring out-of-normal data distributions.
To address the scale variation issue, propose scale adaptation mechanism that learns to softly weight contrastive representation distillation at each scale.
We evaluate the proposed approach three datasets from different imaging modalities: X- ray, MRI, and dermoscopy. Experimental results demonstrate state-of -ott----rrrtrr performance across all datasets.
2

---

Published as a conference paper at ICLR 2025
clean teacher
SAM
scale deseriptor
bottleneck:
Fusion
noisy teacher
res block
block-3
(man elover
block-3
mnvi appat
block-3
conv 1x1
block-2
block-2
block-2
block-1
block-1
block-1
3x3
student
conv 3x3
conv 3x3
simplex noise
size uniformj44 (a, b]
scale adaptation mechanism:
position uniform(e,
GMP
linear
synthetic anomaly
Figure 1: Illustration of the proposed framework during training. It comprises distinct encoding pathways: 1) "clean" teacher encoder followed by a bottleneck, a scale adaptation mechanism, and student decoder, and 2) "noisy" teacher encoder. The two teacher encoders share weights but process different inputs: the clean teacher receives normal data, whereas the noisy teacher processes synthesized anomalies. We employ contrastive reverse distillation by pushing the student's recon- structed features closer to feature representations from the clean teacher and farther from those of the noisy teacher. The scale adaptation module generates spput-specific scale weights used this process.
2 METHODOLOGY
2.1 PRELIMINARY
First, we revisit the original reverse distillation model for anomaly detection as proposed by Deng & Li (2022). It consists of three components: 1) fixed pre-trained encoder serving as the teacher to extract feature maps from an input image; 2) a bottleneck fusing multi-scale features from the encoder into joint representation; and 3) decoder acting the student reconstruct feature maps at each scale from the joint representation.
Let simma, (,, b) a b/ ||||| ||b|| denote the dot product between & normalized and b (i.e., cosine similarity), and uy and Uy feature maps from the k-th layer of the teacher encoder and student decoder, respectively. For knowledge transfer in the model, the following loss is used:
L L=
ΣE sim(us. kk
where Cg is normalization constant, and (h, w) enumerates integral spatial locations in feature maps Uy and After training the teacher encoder and student decoder normal data only, they align to represent normal patterns similar way. For normal test samples that conform to patterns observed during training. the student's representations closely match the teacher's, leading low vector-wize cosine similarity losses. Conversely, for anomalous samples, the student decoder fails properly reconstruct the teacher's feature maps due having learned only normal patterns.
2.2
CONTRASTIVE REVERSE DISTILLATION
2.2.1
NETWORK ARCHITECTURE AND Loss
One limitation of the approach proposed by Deng & Li (0022) is that the teacher only sees normal data, i.., in-distribution examples, during training and does not attempt to explore potential out-of- distribution representations. This may result in lack of discrimination for anomalies in real-worl scenarios. To address this, we propose a novel paradigm called contrastive reverse distillation for anomaly detection (see Figure []. It introduces second teacher encoder, termed "noisy" teacher,
3

---

Published as a conference paper at ICLR 2025
which can synthesize plausible out-of-norman representations to anomalous instances. The resulting model contains two distinct encoding pathways: one a "clean" teacher encoder followed by bottleneck and student decoder described in Section 2.1 and the other is the noisy teacher encoder. The two teacher encoders share weights but take different images as input-the clean teacher sees normal data while the noisy teacher sees synthesized anomalies.
Formally, let a represent a normal training image, x' = g(x)) be a synthesized abnormal image, and refer to the output of the bottleneck, i.e., multi-scale representation of æ. We denote feature maps from the k-th layer of the clean teacher encoder, noisy teacher encoder, and student decoder as u₁ = f(x). 2 f(x'). and vk. respectively, where f()) indicates the encoders. In our model, we aim to push uR and Uy closer while pushing and apart. To this end, we design novel contrastive reverse distillation loss:
- -= zk. V/)) +
1 sim simm (ug VK
where c is small value added to the denominator to avoid division-by-zero errors in practice. It worth noting that unlike Eq- (T). here un and Vú are reshaped into 1D representations prior to calculating the cosine similarity between them, and the same reshaping operation is applied 2ą as well. This makes results more stable.
2.2.2 IMAGE SYNTHESIS FOR NOISY TEACHER
Medical images typically exhibit a power law distribution of frequencies, with lower frequency components dominating the image content (Wyatt et al. 2022). Based on the assumption that both normal and abnormal images adhere to this power law, we are motivated to synthesize abnormal images by applying noise with similar distribution. While such noise Gaussian random fields or engineered covariance matrices, we opt for a simpler approach utilizing simplex noise (Perlin 2002), following methods of Tien et al. (2023) and Wyatt al. (2022). Sim- plex noise enables precise control over the frequency distribution of images. In contrast to Gaussian noise, simplex noise produces smooth, structured randomness, making it well-suited for task. We compare the performance of these two noise types in our model (cf. Section 3.3.3).
be generated through
In our approach, for each training image æ. we first sample noise size and position from uniform distributions. Then, simplex noise (Perlin (2002) is generated with six octaves and a persistence of y 0.6. Finally, the generated noise is added to the image, scaled by a factor of X 0.2. This process synthesizes an abnormal image x', introducing structured perturbations that mimic potential anomalies.
2.3 SCALF-AWARE CONTRASTIVE REVERSE DISTILLATION
To address the scale variation of anomalies in medical images, we propose learning a scale descriptor a [oi,..., a where K is the number of layers, using head h(-). The process involves em- ploying global max pooling (GMP) to spatially shrink Φ. generating channel-wise statistics. Since & can be viewed as the joint representation multi-scale features, its statistics are informative about the image content across scales. Subsequently, a linear layer with softmax normalization maps the channel-wise statistics to the scale descriptor a. This process is formulated as:
h(Φ)) = softmax(W GMP(Φ))
(3)
The final loss is obtained by recalibrating knowledge transfer at different seales with a:: sim (ug, vx))
C
Σ ah sim vx) +
The scale descriptor acts ipput-specific scale weights for contrastive reverse distillation. In this regard, it intrinsically introduces dynamics conditioned on the input, helping to boost the discrim- inability of the model.
2.4 ANOMALY SCORING
During inference, given a test image, we extract a set of feature maps, }u;} and {2x}. from the clean teacher encoder and student decoder, respectively, defined in Section 2.2.1 Besides,
4

---

Published as a conference paper at ICLR 2025
obtain scale descriptor from the scale adaptation module. We compute the vector-wise cosine similarity between in and Uy for each scale k. The resulting similarity map is then weighted by the corresponding ag and upsampled to match the input image resolution. We aggregate weighted similarity maps across all scales to construct comprehensive anomaly map. The final anomaly score for the input image is obtained by computing the mean value of this aggregated anomaly map.
2.5 COMPARISON WITH RD++
recent work closely related to our approach is RD++ (Tien et al., 2023), which also builds upon the reverse distillation paradigm and utilizes synthesized abnormal images. While the proposed model shares some architectural similarities with RD++ including the use of dual encoding path- ways, there are fundamental differences in approach and objectives. A key distinction lies how anomalous information is used. Our method leverages synthesized anomalies to enhance feature discrimination by employing contrastive regularization between the student's reconstructed features and representations from both teachers. In contrast, RD++ introduces multiple regularizations for two encoding pathways, aiming to encourage the model to learn how to reconstruct normal features from pseudo-abnormal regions. This design inherently restricts the flow of anomalous information to the student network. Furthermore, our approach incorporates a scale adaptation mechanism for reverse distillation, addressing the critical issue of scale variation in anomaly detection challenge not explicitly tackled by RD++. We provide a comparison of the two models in Table
EXPERIMENTS
3.1
EXPERIMENTAL SETTINGS
3.1.1
DATASETS
We evaluate our proposed method three widely-used medical imaging datasets: the RSNA Pneu- monia Detection Challenge datase the Brain Tumor MRI datase and the ISIC 2018 datase
RSNA: This chest X-ray dataset comprises 8,851 normal and 6,012 lung opacity images. Follow- ing Cai et al. (2022), ) we utilize 3,851 normal images for training and a balanced test set of 1,000 normal and 1,000 abnormal images.
Brain Tumor: This dataset consists of 2,000 MRI slices without tumors, 1,621 with gliomas, and 1,645 with meningiomas. We categorize glioma and meningioma slices anomalies. The normal instances are sourced from Br35H15 and Saleh et al. (2020), while the anomalous cases are from Saleh et al.) (2020) and Cheng et al. (2015). In line with Cai et al. (2022), our experimental setup includes 1,000 normal slices for training and a test set of 600) normal and 600 abnormal slices (equally split between glioma and meningioma).
This skin lesion dataset, originating from the ISIC 2018 challenge, contains dermoscopic images across seven categories. Consistent with previous studies (Lu & Xu 2018 Guo al. 2024). we designate nevus as the normal class. Our experimental protocol, following Cai et al. 2024 employs 6,705 normal images from the official training set for model training. Our test set comprises 909 normal images and 603 abnormal images (distributed across the remaining six categories) from the official test set.
ISIC:
3.1.2 EVALUATION METRICS
Unsupervised anomaly detection methods typically generate continuous-valuce predictions. There- fore, we use the area under a receiver operating characteristic (ROC) curve (AUC) as our primary nature. Moreover, report F1 score and accu- racy. For these metrics, we determine the optimal threshold based on the best Fl score, following the approach of Zhao et al. (0023).
evaluation metric, given its threshold-independdee
brain 0rain-Bumorrmiii-4411111
https://ww..kage
5

---

Published as a conference paper at ICLR 2025
RSNA
Brain Tumor
ISIC
AUC
F1
ACC
AUC
F1
ACC
AUC
F1
ACC
68.33
67.85
52.90
80.88
84.79
82.33
73.59
64.67
67.20
UAE
84.36
79.47
77.55
94.50
91.38
90.75
74.12
65.91
68.25
MorphAEus
80.87
75.74
72.80
64.68
70.43
60.67
68.30
61.11
57.74
GAN Ensemble
82.10
75.30
74.30
66.60
68.40
64.00
63.70
62.70
54.60
MemAE
68.65
67.95
53.45
79.91
82.63
80.00
73.47
64.67
64.81
PatchCore
86.33
80.86
79.40
93.63
89.23
88.42
68.09
61.98
59.26
SQUID
70.38
72.40
65.95
41.33
66.67
50.00
57.47
54.78
46.11
FastFlow
76.00
73.68
67.95
85.62
80.41
77.58
67.27
63.46
57.34
CFLOW-AD
70.26
70.20
62.05
36.35
66.67
50.00
66.97
60.81
57.80
CutPaste
55.86
66.69
50.05
74.24
67.99
52.92
64.25
57.05
39.95
NSA
82.13
75.87
82.13
83.20
79.00
76.17
69.08
62.53
58.47
RD4AD
84.29
78.09
76.80
90.52
87.24
85.67
76.48
67.94
67.72
RD++
88.00
81.57
81.20
91.68
87.43
85.83
75.47
67.33
67.59
ReContrast
87.53
81.24
80.70
91.67
85.99
84.33
80.02
70.20
75.46
UniAD
73.69
69.71
65.85
62.30
69.92
58.42
73.78
65.48
66.60
SimpleNet
69.06
68.99
62.70
93.93
88.74
88.50
69.01
60.66
56.99
EfficientAD
74.88
73.12
68.20
78.41
76.20
72.00
60.32
56.95
39.81
UCAD
70.89
69.88
62.45
87.42
80.68
80.08
67.88
59.62
55.29
Ours
91.01
84.05
83.40
98.88
96.52
96.50
83.10
72.07
75.60
Table 1: Quantitative comparison the proposed method against state-of-the-att approaches on three public medical anomaly detection datasets. The best results for each dataset and metric highlighted in bold, and the second-best results are underlined
3.1.3 IMPLEMENTATION DETAILS
We conduct all experiments using PyTorch on a single NVIDIA RTX 3090Ti GPU. Our encoder utilizes WideResNet50 (Zagoruyko) 2016) pre-trained ImageNet Russakovsky et al. 2015). We also report the performance of our network using ResNet18 8 (He et al. 2016) and ResNet50 (He al. 2016) the encoder in Section 3.3.4 We resize all images 256 x 256 pixels and apply no data augmentation during training. To train our model, we employ the Adam optimizer (Kingma (0.5, 0.999) and a learning rate of le-3. We train for 4,000 iterations with a batch size of 16. Our decoder mirrors the encoder, identical to that used in RD4AD (Deng & Li 2022). For competing methods, we utilize their publicly available codes and adhere their default training configurations.
& Ba 2015) with
3.2 COMPARISON WITH OF-TE- METHODS
We comprehensively evaluate our approach against existing unsupervised anomaly detection meth- ods. Our comparisons encompass reverse assillationnbbsel methods such RD4AD (Deng Li 2022) and its variants, RD+ (Tien et al. 2023) and ReContrast (Guo et al. 2023), as the proposed model builds upon this paradigm. Given
of synthetic anomalies, we also compare with syn- thetic anomaly-based methods like NSA (Schlüter et al.) 2022) and CutPaste (Li et al. 202T). Fur- thermore, we assess our approach against methods using generative models (AE, UAE (Mao al.) 2020). GAN Ensemble (Han et al. (0211), and MorphAEus (Bercea et al.) 2023)). methods utiliz- ing memory banks (MemAK (Gong al. 2019), PatchCore (Roth al. (002)) and SQUID (Xiang et al. 2023)), and methods leveraging normalizing flows (FastFlow (Yu et al. 2021b and CFLOW. AD (Gudovskiy et al. 2022)) We also include other recent relevant methods such as UniAD (You al. 2022), SimpleNet (Liu et al. 2023). EfficientAD (Batzner et al.) 2024), and UCAD (Liu al. 2024 in our comparison. As shown in Table I the proposed framework demonstrates superior per- formance across all three datasets, consistently outperforming competitors on all evaluation metries. Notably, in terms of the primary metric AUC, our method shows substantial improvements over the second-best methods. On the RSNA dataset, achieve 3.01 % improvement. For the Brain

---

Published as a conference paper at ICLR 2025
RSNA
Brain Tumor
ISIC
CRD
SAM
AUC
F1
ACC
AUC
F1
ACC
AUC
F1
ACC
84.29
78.09
76.80
90.52
87.24
85.67
76.48
67.94
67.72
89.87
83.69
83.20
91.45
88.17
86.67
78.87
69.50
71.56
91.01
84.05
83.40
98.88
96.52
96.50
83.10
72.07
75.60
Table 2: Ablation study quantifying the impact of each component in the proposed method on three datasets. We report the performance of full model, as well as variants with the following components ablated: contrastive reverse distillation (CRD) and scale-adaptive mechanism (SAM).
RSNA
Brain Tumor
ISIC
normal abnormal
normal abnarmal
abnormal
normal
normal abnormal
normal
abnormal
Figure 2: Comparison of anomaly score distributions for normal (blue) and abnormal (red) samples in the test sets across datasets. Top: Distributions obtained from the baseline RD4AD (Deng & Li 2022). Bottom: Distributions generated by our proposed model. Scores are normalized to [0,1] for each subfigure enable direct comparison. Our approach demonstrates enhanced separation, leading to improved anomaly detection performance.
Tumor dataset, the improvement is 4.38%, while on the ISIC dataset, 3.08% increase in performance.
3.3
ABLATION STUDY
We conduct ablation studies validate the efficacy of each component in our framework. Using the RD4AD model (Deng & Li. 2022) our baseline, we progressively incorporate our contributions, including contrastive reverse distillation and the scale-adaptive mechanism. Table / presents the numerical results.
3.3.1 EFFECTIVENESS OF CONTRASTIVE REVERSE DISTILLATION
Augmenting the reverse distillation model with the proposed contrastive student-teacher learning yields significant improvements. On the RSNA dataset. we achieve gains of 5.58%, 5.6%, and 6.4% in AUC, Fl score, and accuracy, respectively, over the baseline. We observe consistent improve- ments across all metrics the Brain Tumor and ISIC datasets well.

---

Published as a conference paper at ICLR 2025
3.3.2 EFFECTIVENESS OF SCALE ADAPTIVE MECHANISM
Further incorporating the scale adaptive mechanism into the contrastive reverse distillation frame- work leads to additional performance enhancements. On the Brain Tumor dataset, observe im- provements of 7.43%, 8.35%, and 9.83% in AUC, Fl score, and accuracy. respectively. The RSNA dataset shows consistent gains (1.14% in AUC, 0.36% in Fl score, and 0.2% accuracy), as does the ISIC dataset (4.23% in AUC, 2.57% in F1 score, and 4.04% in accuracy). These results confirm the importance of addressing scale variation in anomaly detection for medical images.
3.3.3 IMPACT OF NOISE TYPE AND INTENSITY
We evaluate our model's performance using two types of noise for creating synthetic anomalies: simplex noise and Gaussian noise. Table 3 presents the results, indicating that simplex noise gener- ates more natural pseudo-anomalies that better align with anatomical structures in medical imaging compared to Gaussian noise.
We further investigate the impact of varying simplex noise intensities on model performance. Fig- ure 3 shows our method's s performance on the RSNA dataset as increases from 0.1 to 0.5. We observe that the intensity of simulated anomalies significantly influences model performance. Low noise (A = 0.1) appears insufficient to effectively emulate real abnormalities in medical images. The model'ss performance peaks at moderate intensity (A 0.2). When À reaches 0.5, approach's performance drops sharply, suggesting that excessive noise may interfere with key medical features, making it challenging for the model to distinguish between real pathological changes and excessive image distortions.
The results demonstrate that the choice of noise type and intensity plays crucial role in the effec- tiveness of synthetic anomaly generation for improving anomaly detection in the proposed frame- work.
3.3.4
QUANTITATIVE COMPARISON ACROSS DIFFERENT BACKBONES
Table presents the performance of our model with various backbone architectures. Generally, deeper and wider networks exhibit stronger representational capabilities, enhancing our model's ability to detect anomalies more precisely.
3.3.5 QUALITATIVE ANALYSIS
To qualitatively assess the discriminative capability of our framework, we visualize the distributions of anomaly scores for normal and abnormal images within all datasets in Figure / The overlap between normal and abnormal histograms represents samples from different categories that share identical anomaly scores. A smaller overlap indicates a stronger discriminative capability of a model separating normal and abnormal instances. As evident from Figure 2 compared to the baseline model, the proposed framework better separates normal and abnormal images, exhibiting superior anomaly detection performance.
3.4 DISCUSSION
The performance difference between ISIC and Brain Tumor datasets arises from their inherent structural complexities. The ISIC dataset contains multiple dermatological anomalies (melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma, and vascular lesion) with subtle, often indistinct morphological features, challenging model discrimina- tion. In contrast, Brain Tumor images reveal well-defined, high-contrast lesions with clear structural boundaries, rendering anomaly detection more straightforward for models.
The Brain Tumor MRI dataset features highly variable tumor sizes across slices, making the SAM module more impactful, whereas the RSNA X-ray dataset exhibits more consistent anomaly sizes, thus limiting SAM's comparative effectivenesss. This differential performance aligns with the un- derlying dataset characteristics and underscores the importance of scale adaptability across diverse medical imaging contexts.
8

---

Published as a conference paper at ICLR 2025
RSNA F1
Brain Tumor
ISIC F1
Noise Type
AUC
ACC
AUC
F1
ACC
AUC
ACC
Gaussian Simplex
90.70
84.11 84.05
84.05
75.23
74.88
68.25
74.30 83.10
66.20 72.07
67.99
91.01
83,40
98.88
96.52
96.50
75.60
Table Comparative analysis of model performance under various noise types on all datasets.
0.1
0.2
0.3
0.4
0.5
84
82
80
78
AUC
F1
ACC
Figure 3: Effect of À model performance the RSNA dataset. À ranges from 0.1 to 0.5 in 0.1 increments, with higher values corresponding to increased simplex noise levels. Performance metries (AUC, Fl, and ACC) are shown as À increases left right.
RELATED WORK
Reconstruction-basee methods have emerged prominent approach in unsupervised anomaly detection. [Schlegl et al. (2017) pioneer the use of GANs for this purpose with AnoGAN, later introducing f-AnoGAN (Schlegl al. 2019), faster variant employing an encoder to map images to latent space. In addition, various autoencoder architectures are explored, including variational autoencoder (Zimmerer et al.) (018)) and vetor--uantized variational autoencoder (Naval Marimont & Tarroni 2021). To address the overgeneralization problem. Soonlm,Goonaaaa et al. (2019) propose memory- augmented autoencoder. Given an input image, they first obtain an encoded representation from the encoder and then use it as a query to retrieve the most relevant memory items for reconstruction. Park ict al. (2020) exploit a memory module to record prototypical patterns of normal instances. Several works (Rudolph et al. 2021 Gudovskiy et al. 2022 Yu et al. 2021) leverage normalizing flows, enabling exact likelihood estimation for image modeling, and achieve good performance in anomaly detection. For example, Gudovskiy et al. (2022) use conditional normalizing flow framework, and Yu et al. (2021) propose a 2D normalizing flow model for unsupervised anomaly detection and localization.
Self-supervised learning (Jing & Tian. 2021 has also been applied to anomaly detection. Some works train models detect synthetic anomalies and directly apply them real abnormalities. To make synthesized anomaly images more natural, Tan et al. (2021) and Schlüter et al. (2022) integrate Poisson image editing seamlessly blend scaled patches of various sizes from separate images. This creates a wide range of synthetic anomalies that are more similar to natural abnormalities than pre- vious data augmentation strategies for supervised anomaly detection. Furthermore, two-stage approaches are studied. For instance, et al. (202T) propose simple strategy to generate synthetic anomalies for anomaly detection by cutting an image patch and pasting a random location of large image. They then learn self-supervised representations by classifying normal and sythesized abnormal data. Finally, a generative one-class classifier is built on the learned representations. Sohn al. (202T) learn self-supervised representations from normal data by solving proxy tasks, e.g., rotation prediction and contrastive learning, and then train one-class classifiers using the learned representations.
Knowledge distillation from pre- treitrd models showcases promising results recently (Salehi et al. 2021 Deng & [i 2022 Batzner et al. 2024). Salehi al. (2021) propose to the distillation

---

Published as a conference paper at ICLR 2025
RSNA
Brain Tumor
ISIC
Backbone
AUC
F1
ACC
AUC
F1
ACC
AUC
F1
ACC
Resnet-34
85.20
78.96
77.25
92.17
87.95
86.67
78.76
70.43
69.44
Resnet-50
88.45
82.15
81.25
94.16
89.81
89.00
80.83
71.28
71.96
Wide ResNet-50
91.01
84.05
83.40
98.88
96.52
96.50
83.10
72.07
75.60
Table 4: Quantitative performance comparison of multiple backbone architectures for all datasets.
of features at various layers of an expert network, pre-trained on ImageNet, into a simpler cloner network. They detect anomalies using the discrepancy between the expert and cloner networks' intermediate feature maps given an input image. Deng & Li (2022) devise reverse distillation
paradigm, which is further explored in subsequent works (Deng Li 2022 Tien et al. 2023).
Recently, You et al. 2002 formulate universal anomaly detection and propose Transformer-base feature reconstruction model using layer-wise query decoder to model complex multi-class normal
distributions.
5
CONCLUSION
This paper presents novel scale-aware contrastive reverse distillation model for unsupervised anomaly detection. Our approach introduces a contrastive student-teacher framework comprising clean teacher encoder, noisy teacher encoder, and student decoder, coupled with scale adap- tation mechanism. This architecture enables our model to derive robust feature representations and effectively address the scale variation issue inherent in anomalies. Extensive experiments on bench- mark datasets demonstrate that the proposed method achieves state-of-the-art performance, under- scoring its efficacy in unsupervised anomaly detection tasks. Our current approach relies random spatial sampling for noise generation. Future research directions include incorporating modality- specific anatomical priors for anomaly localization in medical imaging. The exploration of learn- able noise like (Cai & Fan, 2022) or adaptive hybrid noise generation techniques presents another promising direction. These extensions would enhance the realism of synthetic anomalies toward improved model performance.
REFERENCES
Kilian Batzner, Lars Heckler, and Rebecca König. EfficientAD: Accurate visual anomaly detection at millisecond-level latencies. In WACV, 2024.
Cosmin 1. Bercea, Daniel Rueckert, and Julia A. Schnabel. What do AEs learn? Challenging common assumptions in unsupervised anomaly detection. In MICCAI, 2023.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. CVPR, 2020.
Jinyu Cai and Jicong Fan. Perturbation learning based anomaly detection. In NewrIPS, 2022.
Yu Cai, Hao Chen, Xin Yang, Yu Zhou, and Kwang- Ting Cheng. Dual-distribution discrepancy for anomaly detection in chest X- X-Rays. In MICCAI, 2022.
Yu Cai, Hao Chen, Xin Yang, Yu Zhou, and K Kwang- Ting Cheng. Dual-distribution discrepancy with self-supervised refinement for anomaly detection in medical images. Medical Image Analysis, 86: 102794, 2023.
Yu Cai, Weiwen Zhang, Hao Chen, and Kwang-Ting Cheng. MedlAnomaly: A comparative study of anomaly detection in medical images. arXiv preprint 244i::2404.045166 2024.
Jun Cheng. Wei Huang, Shuangliang Cao. Ru Yang, Wei Yang. Zhaoqiang Yun, Zhijian Wang, and Qianjin Feng. Enhanced performance of brain tumor classification via tumor region augmentation and partition. Public Library of Science One, 10110):0140381, 2015.
10

---

Published as a conference paper at ICLR 2025
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one- class embedding. In CVPR, 2022.
Dong Gong, Linggiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality detect anomaly: Memory-augment deep autoencoder for unsupervised anomaly detection. In ICCV, 2019.
Denis A. Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. CFLOW-AD: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In WACV, 2022.
Jia Guo, Shuai Lu, Lize Jia, Weihang Zhang, and Huiqi Li. ReContrast: Domain-specific anomaly detection via contrastive reconstruction. NeurIPS, 2023.
Jia Guo, Shuai Lu, Lize Jia, Weihang Zhang, and Huiqi Li. Encoder-decoder contrast for unsu- pervised anomaly detection in medical images. Institute of Electrical and Electronics Engineers Transactions on Medical Imaging, 112-)1112 2024.
Changhee Han, Leonardo Rundo, Kohei Murao, Tomoyuki Noguchi, Yuki Shimahara, Zoltán Ádám Milacski, Saori Koshino, Evis Sala, Hideki Nakayama, and Shin ichi Satoh. MADGAN: Unsu- pervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction. BioMed Central bioinformatics Bioinformatics, 22-S(2):31, 2021a.
Xu Han, Xiaohui Chen, and Li-Ping Liu. GAN ensemble for anomaly detection. In AAAI, 202
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Wenqian Jiang, Yang Hong, Beitong Zhou, Xin He, and Cheng Cheng. A GAN-based anomaly detection approach for imbalanced industrial time series. Institute of Electrical and Electronics Engineers Access, 7:143608 143619, 2019.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. Institute of Electrical and Electronics Engineers Transactions on Puttern Analysis and Machine Intelligence, 43(1 1):4037-4058, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. CutPaste: Self-supervised learning for anomaly detection and localization. In CVPR, 2021.
Jiaqi Liu, Kai Wu, Qiang Nie, Ying Chen, Bin-Bin Gao, Yong Liu, Jinbao Wang, Chengjie Wang, and Feng Zheng. Unsupervised continual anomaly detection with contrastively-leamee prompt. In AAAI, 2024.
Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. SimpleNet: A simple network for image anomaly detection and localization. In CVPR, 2023.
Yuchen Lu and Peng Xu. Anomaly detection for skin discase images using variational autoencoder. arXiv preprint 1rXiv:1807.013:2 2018
Yifan Mao, Feifei Xue, Ruixuan Wang, Jianguo Zhang, Wei-Shi Zheng, and Hongmei Liu. Abnor- mality detection in chest X-Ray images using uncertainty prediction autoencoders. In MICCAI, 2020.
Sergio Naval Marimont and Giacomo Tarroni. Anomaly detection through latent space restoration using vector quantized variational autoencoders. In ISBI, 2021.
Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided normality for anomaly detection. In CVPR, 2020.
Ken Perlin. Improving noise. SIGGRAPH, 2002.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter V. Gehler. Towards total recall in industrial anomaly detection. In CVPR, 2022.
11

---

Published as a conference paper at ICLR 2025
Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but DifferNet: Semi-supervise defect detection with normalizing flows. In WACV, 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11:2211-222, 2015.
Ahmad Saleh, Rozana Sukaik, and Samy S Abu-Naser. Brain tumor classification using deep learn- ing. In iCareTech, 2020.
Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H. Rohban, and Hamid R. Rabiee. Multiresolution knowledge distillation for anomaly detection. In CVPR, 2021.
Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In IPMI, 2017.
Thomas Schlegl, Philipp Sceböck. Sebastian M. Waldstein, Georg Langs, and Ursula Schmidt- Erfurth. f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis, 54:30 44, 2019.
Hannah M. Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz. Natural synthetic anomalies for sulf-supervised anomaly detection and localization. In ECCV, 2022.
Nina Shvetsova, Bart Bakker, Irina Fedulova, Heinrich Schulz, and Dmitry V. Dylov. Anomaly detection in medical imaging with deep perceptual autoencoders. Institute of Electrical and Elec- tronics Engineers Access, 1:17111--18888 2021.
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister, Learning and evaluating representations for deep one-class classification. In ICLR, 2021.
Jeremy Tan, Benjamin Hou, Thomas G. Day, John M. Simpson, Daniel Rueckert, and Bernhard Kainz. Detecting outliers with poisson image interpolation. In MICCAI, 2021.
Tran Dinh Tien, Anh Tuan Nguyen, Nguyen Hoang Tran, Ta Duc Huy, Soan Thi Minh Duong, Chanh D. Tr. Nguyen, and Steven Q. H. Truong. Revisiting reverse distillation for anomaly detection. In CVPR, 2023.
Guodong Wang, Shumin Han, Errui Ding, and Di Huang. Student-teacher feature pyramid matching for anomaly detection. In BMVC, 2021.
Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks. AnoDDPM: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In CVPR Workshops, 2022.
Tiange Xiang, Yixiao Zhang, Yongyi Lu, Alan L. Yuille, Chaoyi Zhang, Weidong Cai, and Zongwei Zhou. SQUID: Deep feature in-painting for unsupervised anomaly detection. In CVPR, 2023.
Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. In NeurlPS, 2022.
Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. FastFlow: Unsupervised anomaly detection and localization via 2D normalizing flows. arXiv preprint arXiv:2111.07677. 2021.
Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv:1605.07144, 2016.
Yuzhong Zhao, Qiaoqiao Ding, and Xiaoqun Zhang. AE-FLOW: Autoencoders with normalizing flows for medical images anomaly detection. In ICLR, 2023.
David Zimmerer, Simon AA Kohl, Jens Petersen, Fabian Isensee, and Klaus H Maier-Hein. Context-encoding variational autoencoder for unsupervised anomaly detection. arXiv preprint arXiv:1812.05944,, 2018
12

---

Published as a conference paper at ICLR 2025
APPENDIX
Algorithm. We present detailed procedures for synthesizing abnormal images and training our model Algorithm | and Algorithm respectively.
Algorithm Synthesize abnormal images
is the height of the input image.
w is the width of the input image.
[a. b] represents the set {a R:oaza for epoch = to n do
for m, normal training set do
Get Anoiss C [10, (h,8)8
Get Wnoise S [10, (n/
Get Istart, Mstart S [0, Knoise]; [0, Wnoise
Randomly generate simplex noise:
Simplex(( Simplex((hnoise: Wnoise),.N. N 6,7 0.6)
& = zeros(h, w)
Fend Istart Annois
Vend = Metars + Wnosse
E[Tstart Tend: Yetart yend] =
x D2 + x * (A: the intensity of the added noise)
Training process
end for end for
Algorithm 2 Pseudo-code of our approach in one epoch training
E, D: Encoder, Scale Adaptation Module, Bottleneck, Decoder Ug Zg. Normal and synthetic anomalous features at block Optimizer Adam
Load a mini-batch of normal and pseudo-abnormal samples
for x, x in dain-dattl do
Get encoder outputs for normal and pseudo-abnormal images at three blocks
u1, 22, u3 (xx)
21, 22, {xx)
Get decoder outputs
11 U2
D(B(u), u2 U3))
a = S(B( U11 U2 ua))
L1 = Ex=
11 0+(1 sim u4, 0z))
Le Σ
ax(1 sim vn))
= L2
b.backward Optimizer.step
end for
Inference Time and Paramters. We compare our model with competing methods in terms of AUC, inference time, and memory usage at inference (see Figure 4 The focus the inference phase is particularly relevant due to its critical importance in clinical applications, where real-time processing and memory efficiency directly impact the feasibility and deployment potential of model. The advantages of our model in these aspects make it a promising approach for practical applications. We additionally report trainable parameter counts for our model and competing meth- ods in Table
ROC Curves. We visualize the ROC curves to compare our method with the top-performing base-
lines across all datasets (see Figures 5
13

---

Published as a conference paper at ICLR 2025
Comparisons of Anomaly Detection Methods
90
Ours
RD++
AcContrus:
Par Core
RD4AD
AUROC (%)
NSA
GAN Ensemble
20 30
40
50
60
70
80
Inference Time (ms)
Figure 4: Performance comparison of anomaly detection methods. Evaluation metrics include: AUROC (vertical axis), inference time (horizontal axis), and memory footprint (circle radius). Our method achieves statc-of-the-art performance, delivering the highest AUROC while demonstrating superior computational efficiency. Specifically, our approach 6x faster than PatchCore, 4x faster than GAN Ensemble, 3x faster than CFlow, and 2x faster than NSA.
Ours
RD4AD
RD+
ReContrast
CFlow
GAN Ensemble
PatchCore
UAE
NSA
AUC
91.01
84.29
88.00
87.53
70.26 288.5
82.10
86.33
84.36
82.13
Params (MB)
264,4
263.5
272.8
527.9
381.46
275.6
256.3
271.2
Table Comparative performance of anomaly detection methods on the RSNA dataset: AUC scores and model complexity. Our approach achieves the highest AUC with more parameter-efffcient design.
Receiver Operating Characteristic (ROC) Curve
0.8
True Positive Rate
0.4
Ours (AUC- (AUC-91.01)
RD4AD (AUC=84.29)
(AJC=68.00)
UAE (AUC-84.36)
ReCantrast (AUC-87.53)
(AUC=86.33)
0.0
0.0
0.2
D.4
0.6
0.8
1.0
Rate
Figure 5: Comparison of ROC curves between our method and the top anomaly detection methods on the RSNA dataset.
14

---

Published as a conference paper at ICLR 2025
Receiver Operating Characteristic (ROC) Curve
1.0
True Pesitive Rate
0.6
(AUC-98.88)
RD4AD (AUC-90.52)
0.2
RD++ (AUC=91688)
(AUC-94.50)
PatchCore (AUC-93.63)
Simpletett (AUC=93.93)
False Positive Rate
Figure 6: Comparison of ROC curves between our method and the top anomaly detection methods on the Brain Tumor dataset.
Receiver Operating Characteristic (ROC) Curve
1.0
True Posilive Rate
0.6
Ours (AUC=83.10)
0.2
RD4AD (AUC=76.48)
(AUC-75.47)
UAE (AUC-74.12)
ReContrast (AUC=80.02)
False Positive Rate
Figure 7: Comparison of ROC curves between our method and the top anomaly detection methods on the ISIC dataset.
15


Reviews:
[{"review_id": "k0qHXRnLeT", "reviewer": "Reviewer_W919", "summary": "The paper presents a scale-aware contrastive reverse distillation model for unsupervised anomaly detection, focusing on improving performance in fields like medical imaging, where labeled anomalous data is often scarce. Building on this, the authors introduce a contrastive student-teacher learning approach that generates out-of-normal distributions to enhance feature discriminability. Additionally, they propose a scale adaptation mechanism that adjusts contrastive distillation losses at various scales to account for variations in anomaly size, addressing limitations in existing methods that struggle with scale variation and feature discrimination. Evaluation on benchmark datasets shows that this model achieves reasonable performance, validating its effectiveness.", "questions": "a) Anomaly detection is an important topic where identifying unusual patterns can have a direct impact on various applications. In this sense, this paper researches a domain-specific area in anomaly detection, i.e., medical images, using reverse-distillation. \n\nb) While this paper provides valuable refinements in the area of anomaly detection, particularly through its scale-aware contrastive reverse distillation model, the novelty appears somewhat incremental. Firstly, the study primarily builds on and combines well-established methods, reverse distillation and contrastive learning, rather than introducing a fundamentally new approach. Secondly, the approach to enhance feature discriminability by generating synthetic or perturbed anomaly data is not novel; similar methods have been previously explored, such as in [1]. The effectiveness of this technique can also vary significantly depending on the choice of noise or perturbation (rather than learnable in [1]), which may limit its reliability. Thirdly, fusing multi-scale features is a long-standing topic in computer vision, which further contributes to the incremental nature of the contribution. While these refinements have practical value, they represent an evolution of existing methods rather than a new innovation in anomaly detection.\n\n[1] Cai, Jinyu, and Jicong Fan. \"Perturbation learning based anomaly detection.\" Advances in Neural Information Processing Systems 35 (2022): 14317-14330.\n\nc) The evaluation of the proposed method is performed on three medical datasets, it is not clear how good it can be generalized to other domains?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+) Anomaly detection is an important topic\n\n+) The paper is well-structured and easy to follow", "weaknesses": "-) Novelty of this work seems incremental\n\n-) The domain is limited to medical image analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jWqpS505D1", "reviewer": "Reviewer_MtA3", "summary": "This paper introduces a twofold approach: on one hand, data augmentation for unlabeled information is achieved by injecting noise, which then serves as the basis for contrastive learning. On the other hand, parameters are extracted at each layer to achieve multi-scale information fusion, enabling the model to better distinguish between positive and negative samples.", "questions": "1. In the noise generation process, the intensity of each position is sampled from a uniform distribution. However, in real-world medical imaging, anomalous regions in disease cases appear in specific locations; for example, abnormalities in a chest image typically do not appear outside body regions. For instance, you could suggest incorporating prior knowledge about typical anomaly locations in different types of medical images.\n\n2. Additionally, is each test sample measured at the image level or at the patch level? If it is at the image level, the positive-to-negative anomaly ratio in the dataset is relatively high, which does not quite align with the extremely imbalanced nature of anomaly detection tasks.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The methodology section of this paper is clearly written and well-presented.", "weaknesses": "1. In terms of innovation, this article presents two components that are commonly seen in other works. A straightforward suggestion would be for the authors to explore a more adaptable data augmentation method to address the issue raised in “Question 1.”\n\n2. Regarding the writing, the **Introduction** section of this article is somewhat confusing. Firstly, the paragraphing is suboptimal. In the second paragraph of the introduction, the author discusses the shortcomings of anomaly detection models based on generation. The third paragraph explains existing work aimed at improving these models, followed by an exploration of the development of self-supervised learning. Although self-supervised learning indeed enriches unsupervised learning within deep learning, the third paragraph would be clearer if it unified these two parts, focusing on how self-supervised learning specifically addresses the issues raised in the second paragraph. Secondly, and more importantly, an excessive portion of the introduction is dedicated to discussing related work, lacking sufficient focus on the contributions of this paper. Maybe you could merging the discussion of existing work and self-supervised learning, followed by a clear transition to the paper's contributions. Additionally, you could recommend what key points about the paper's contributions should be emphasized earlier in the introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5w6PNvFUzA", "reviewer": "Reviewer_tPrf", "summary": "Two issues arise in reverse knowledge distillation for medical imaging, an inability to properly distinguish between different features and an inability to deal with different scales. This work proposes two methods to address these issues: a contrastive student-teacher learning approach that involves using both a “good” and a “bad” teacher, as well as a scale adaptation mechanism. Results demonstrate that on almost every dataset the proposed approach is able to outperform existing techniques.", "questions": "Q1: Please address each of my issues 1-5. If they are adequately addressed I would be willing to increase my score for the paper accordingly.\n\nEDIT: The reviewers have address my concerns adequately. I have update the score of my review accordingly.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well written and the ideas are easy to follow. I appreciate how the authors identify the two problems facing medical imaging in this domain and then propose solutions. The new approach outperforms all other methods that the authors compare with in table 1.", "weaknesses": "Issue 1: This paper doesn’t have any formal proofs or theorems to guarantee its effectiveness, therefore the strengths of the paper must rely on the experiments and empirical results. For the datasets I will admit as a review I am not very familiar with them. For example, on the image dataset CIFAR-10, a few years ago a jump from 93% to 98% was considered significant. Today an increase from 99% to 99.9% on CIFAR-10 would not even be considered a major contribution. Likewise, for these datasets it is hard to determine whether a jump of 3.01% and 4.38% (as the authors show) is worthy of publication. Could the authors please clarify how much other papers pushed the accuracy forward in other works, as compared to the increase their method is offering?\n\nIssue 2: The authors say they will release their code upon publication, but I did not see any code given in the supplemental material. This might be an oversight on my part, but where is the code? I would like to see a good faith gesture of releasing an anonymous github for the code or at least a zip file of the code. Too many papers claim SOTA results and never release code. In this day and age it is simply unacceptable to continue this trend. \n\nIssue 3: The authors also don’t mention anything about the runtime of their algorithm. How does the student-teacher setup and training time compare to other existing methods in the field? While this is not a HUGE concern, I think it would be good to have some discussion and analysis of this. For example could you show the training time complexity of your algorithm and the next two best methods in the field? Or even an experimental runtime of the training in seconds of your algorithm and the next two best methods? Without this I feel we don’t have the complete story when comparing methods. \n\nIssue 4: Please see my minor comments about uncited claims in the papers. For acceptance I would like to see citations for all the claims you have made. \n\nIssue 5: I don’t like the terminology “good” and “bad” teacher. It is very non-technical and hard to follow. Can you please change the terminology in your paper and name the teachers more appropriately? \n\nMinor Comments:\n\n=Line 36, “anomalous samples is often prohibitively expensive and time-consuming…” I believe this is correct but I want to see a citation to backup this claim?\n\n=Line 45, “they tend to generalize too well, thereby risking the reconstruction of abnormal regions.” Citation for this claim? \n\n=Table 1 has many entries that are unsorted (aside from the bolded number indicating the best method). It would be better if this table was sorted in order of increasing AUC for the methods.\n\n=Line 167 “Medical images typically exhibit a power law distribution of frequencies, with lower frequency components dominating the image content.” Please give a citation for this bold claim?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2Poe2FUL7", "reviewer": "Reviewer_8DJE", "summary": "The authors propose a new method for anomaly detection in medical data setting using contrastive reverse distillation. The method uses knowledge distillation as it's foundation, implementing pre-trained teachers models along with a student model trained to either minimize or maximize its distance from the teachers in a feature-map space. They additionally utilize a scale adaptive mechanism to allow for better generalizations across different sizes of anomaly.\n\nTheir method's specific construction utilizes two teacher models with the same weights, one \"bad\" and one \"good\". The good model is given benign images while the bad model is given images with synthetic anomalies. The student model is then trained to maximize its cosine similarity with the per-layer features of the good teacher while minimizing the cosine similarity with the bad teacher. At evaluation time anomaly scores are computed by comparing the scaled, per-layer feature spaces of the good teacher against those of the student. Inputs with high levels of dissimilarity between the good model and student are given high anomaly scores.\n\nTheir evaluation against prior work is very comprehensive, comparing against a total of 18 other detectors. They out perform all other detectors on all three datasets (lung, brain, and skin anomalies) in terms of their main metrics (AUC, F1, and Accuracy). The authors further perform ablations with respect to their main contributions of Contrastive Reverse Distilation and Scale Adaptive Mechanisms.", "questions": "Here I include some questions for the authors to consider.\n\n* Why does ISIC seem like the hardest dataset for all the detectors? Why is Brain Tumor the easiest?\n* Why is SAM more impactful in the Brain Tumor dataset while it seems minimally impactful in the RSNA dataset in table 2? Is this related at all to the results of table 3 - where Gaussian noise performs similar to simplex noise on RSNA, but much worse on Brain Tumor?\n* Can the method achieve better generalization by using a mixture of synthetic noise types, perhaps 10% gaussian, 90% simplex? \n* Can we improve performance by including some real world data in the training set, or will that harm generalization?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Anomaly detection is somewhat outside my area of expertise, but the paper seems to frame itself well within the current body of literature. \n* The paper is well written and easy to follow. Figures are also informative and easy to understand.\n* The idea of contrastive reverse distillation seems novel (to the best of my knowledge) and intuitive. I'd like to see how future works can iterate upon this framework in other settings or with improved constructions.\n* Motivation is well established for the usage synthetic anomalies, and the choice of simplex noise is clever.\n* Empirical analysis is very comprehensive in terms of datasets, baselines, and ablations.\n* The proposed method out-performs all baselines in terms of AUC, F1, and Accuracy.\n* The proposed method scales well with model architecture size, which is a good sign that the method is sensible.", "weaknesses": "In general, within my somewhat limited knowledge of this sub-field, I like this paper. In spite of that I think there are a few things the authors may want to consider to further strengthen their work.\n\nI think when discussing detection and AUC scores it is important to give them context within the given domain. See section B of [1] for some example discussion on this topic from a different field. I'd like the authors to consider the following questions:\n\n* Is it more important to maximize the true positive rate in spite of inducing some false positives? \n* Is it better to minimize false positives to increase confidence in the case of a positive prediction?\n* How do you anticipate doctors using these tools in their analysis?\n\nGiven this, it would also be nice if the authors could plot some of the ROCs between their method and some of the baselines (maybe including all of them is too visually messy, but at least the top 5). This way people can compare their method to the baselines at different levels of false positive rate.\n\nI think it would be good if the authors could include some extended discussion, either in the main body or the supplementary material, analyzing the more subtle features of their results. See the next section of my review \"Questions\" for some examples of what I think they can further analyze.\n\nLastly, the results of table 1 are good, but a bit overwhelming. I think it would help if the second best score in each metric is underlined and if the methods are visually grouped based on detector type (e.g. RD4AD, RD++, and ReContrast could be grouped together). Optionally it would be nice if all experiments were averaged over a few runs and given standard deviations or $\\pm$ scores, at least in the supplementary material. I understand if these experiments are too extensive to replicate too many times, though.\n\n [1] \"Membership Inference Attacks From First Principles\" https://arxiv.org/abs/2112.03570", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)