Dual Distillation for Few-Shot Anomaly Detection
Authors: Le Dong
Qinzhong Tan
Chunlei Li
Jingliang Hu
Yilei Shi
Weisheng Dong
Xiao Xiang Zhu
Lichao Mou
Venue: ICLR 2026 Poster (2026)
Keywords: anomaly detection
few-shot learning
knowledge distillation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Anomaly detection is a critical task in computer vision with profound implications for medical imaging, where identifying pathologies early can directly impact patient outcomes. While recent unsupervised anomaly detection approaches show promise, they require substantial normal training data and struggle to generalize across anatomical contexts. We introduce D$^2$4FAD, a novel dual distillation framework for few-shot anomaly detection that identifies anomalies in previously unseen tasks using only a small number of normal reference images. Our approach leverages a pre-trained encoder as a teacher network to extract multi-scale features from both support and query images, while a student decoder learns to distill knowledge from the teacher on query images and self-distill on support images. We further propose a learn-to-weight mechanism that dynamically assesses the reference value of each support image conditioned on the query, optimizing anomaly detection performance. To evaluate our method, we curate a comprehensive benchmark dataset comprising 13,084 images across four organs, four imaging modalities, and five disease categories. Extensive experiments demonstrate that D$^2$4FAD significantly outperforms existing approaches, establishing a new state-of-the-art in few-shot medical anomaly detection. Code is available at https://github.com/ttttqz/D24FAD.

Published as a conference paper at ICLR 2026
DUAL DISTILLATION FOR FEW-SHOT ANOMALY DE- TECTION
Le Dong', Qinzhong Tan i*, Chunlei Li", Jingliang Hu², Yilei Shi?, Weisheng Dong', Xiao Xiang Zhu", and Lichao Mou 2.1
Xidian University, China
2 MedAA Technology (Wuxi) Co. Ltd., China
lichao. mou @med ieteedimainnngn magingai. com
Technical University of Munich, Germany
ABSTRACT
Anomaly detection is a critical task in computer vision with profound implications for medical imaging, where identifying pathologies early can directly impact patient outcomes. While recent unsupervised anomaly detection approaches show promise, they require substantial normal training data and struggle generalize across anatomical contexts. We introduce D²4FAD, a novel dual distillation framework for few-shot anomaly detection that identifies anomalies in previously unseen tasks using only small number of normal reference images. Our approach leverages a pre-trained encoder as a teacher network to extract multi-scale features from both support and query images, while student decoder learns to distill knowledge from the teacher on query images and self-distill on support images. We further propose torn-ttowweigh mechanism that dynamically assesses the reference value each support image conditioned on the query, optimizing anomaly detection performance. To evaluate our method, we curate comprehensive benchmark dataset comprising 13,084 images across four organs, four imaging modalities, and five disease categories. Extensive experiments demonstrate that 4 significantly outperforms existing approaches, establishing a new state-of-the-art in few-shot medical anomaly detection. Code is available at /ttt::://tthub ccmmttttgz/D22/
INTRODUCTION
The automatic detection of anomalies in medical images is crucial yet challenging task in clinical practice. Finding abnormalities early through automated screening enables timely intervention, leading to improved patient outcomes. However, developing robust algorithms for this task remains difficult due to the diverse manifestations of anatomical and pathological abnormalities in different patients. Furthermore, obtaining annotated datasets of verified anomalies is prohibitively expensive and time-consuming in medical contexts. These challenges have driven significant research interest in unsupervised anomaly detection methods, which can identify anomalies without requiring labeled abnormal training data.
Recent years have witnessed substantial advances in unsupervised anomaly detection approaches. Generative models, including autoencoders (Shvetsova et al.) (021) and generative adversarial networks (GANs) (Jiang al. 2019), have shown promising results by learning to reconstruct normal patterns and identifying anomalies through reconstruction errors. Various architectural innovations have further improved detection performance, such as memory banks for storing and retrieving prototypical normal patterns (Gong et al. 2019), normalizing flows to model complex data distributions (Yu et al. 202T), and self-supervised learning for better feature representations (Li et al. 2021). Knowledge distillation has also emerged an effective paradigm (Deng and Li 2022
Work done during an interssiip MedAl Technology (Wuxi) Co. Ltd.
Corresponding author.

---

Published as a conference paper at ICLR 2026
where student network learns to emulate teacher's behavior on normal samples, allowing anomaly detection through discrepancy analysis.
These approaches, however, typically require large amounts of normal training data from target domains data that may not be readily available in many clinical scenarios (Pachetti and Colantonio 2024). In addition, they often struggle generalize across different anomaly detection tasks, limiting their practical utility (Fernando et al., [022]] Cai et al. 2025). This highlights the need for few-shot anomaly detection, where models can adapt new anatomical contexts using only small number of normal reference images. Critically, these models are evaluated on novel tasks unseen during training assess their ability to generalize across different anatomical structures and imaging conditions. In this paradigm, both training and inference are performed in an episodic manner, with each episode containing a support set of few normal images and a query image to be evaluated. This approach is particularly relevant in clinical practice, where physicians often have access to limited normal reference cases when examining new patients or encountering rare conditions. Despite its practical importance, few-shot anomaly detection in medical imaging remains relatively unexplored in the literature.
To address this important yet underexplored problem, we propose D²4FAD, dual distillation framework for few-shot anomaly detection. Our approach leverages a pre-trained encoder as a teacher network to extract multi-scale feature maps from support and query images through parallel pathways with shared weights. student decoder, trained exclusively on normal samples, learns to distill knowledge from the teacher network query images while self-distilling on support images. This design enables anomaly detection at inference time by analyzing feature discrepancies between query and support representations, thereby using normal reference images basis for anomaly identification. Note that some works refer to this process as feature reconstruction. In this work, for consistency with prior anomaly detection methods (Salehi al. 202T, Deng and Li,, 2022] Gu et al.) 2023
Ma et al., 2023) Zhou et al., (2022), we use the term distillation. Furthermore, we introduce a weeight mechanism that dynamically assigns importance to each support image, optimizing their reference values for different query images.
To facilitate research in this direction, we curate a comprehensive medical image dataset comprising four organs, four imaging modalities, and five disease categories, totaling 13,084 images. This dataset provides a standardized benchmark for evaluating few-shot anomaly detection methods across diverse medical contexts.
Our key contributions are:
We formalize the few-shot medical anomaly detection task and establish benchmark dataset spanning multiple organs, modalities, and pathologies.
We propose dual distillation framework that effectively leverages limited normal reference images through knowledge distillation for anomaly detection.
We propose learn-to-weighh weight mechanism that evaluates the reference value of each support image conditioned on query images to enhance anomaly detection performance.
We demonstrate through extensive experiments that our approach achieves state-of-the-art results, significantly outperforming existing methods.
METHODOLOGY
2.1 PROBLEM DEFINITION
We formulate few-shot medical anomaly detection as follows. Given training set comprising several distinct tasks, each containing exclusively normal samples, we aim to develop an anomaly detection model with strong generalization capabilities for unseen anomaly detection tasks. During training, following the few-shot lcarning paradigm, we select one normal image as a query and several fixed normal images from the identical task support samples to facilitate the learning of normal patterns. This task-agnostic model is not optimized for specific tasks but instead learns a universal anomaly detection strategy applicable across diverse medical images.
During inference, our test set contains both normal and abnormal images from novel target tasks unseen during training. We select K normal samples from each target task to form support set,
2

---

Published as a conference paper at ICLR 2026
TRAINING
Query Image
Queery Image
Query Data
Teacher
Support Data
Support Imsages
Support Images
frezen Jayers
learnable layers
teacher-studeat distillation
student self-distllBation
feature similarity
Figure I: Overview our dual distillation framework for few-shot anomaly detection. The archi- tecture incorporates frozen pre- pre-trained teacher encoder and a learnable student decoder. During training, the teacher encoder processes both query and support images, while the student learns to reconstruct multi-scale feature representations through proposed dual distillation approach. At inference time, for previously unseen tasks, anomalies are identified by analyzing discrepancies between query and support image features in the student network. In addition, introduce learn-to-weight mechanism that enhances model performance by dynamically assessing the reference value of each support image relative to a specific query (ef. Section 2.3).
where K E 42,4, 8 in our experiments. This support set enables the model to adapt to previously unseen tasks when determining whether test query image contains anomalies.
2.2 DUAL DISTILLATION
Figure Secureeiilure [illustrates our proposed dual distillation framework. We prioritize architectural simplicity, with the network comprising only two components: a fixed pre-trained encoder and learnable decoder. The encoder, functioning as teacher, extracts visual features from both query and support images, while the decoder, acting student, reconstructs multi-scale feature representations from the encoder's output. We introduce two distillation strategies strategiss-tcacher-sttii tcacher-student distillation and student self-distillation that work in tandem for few-shot anomaly detection. The following subsections elaborate on these strategies.
Teacher-Student Distillation We leverage a pre-trained encoder as the teacher network, which has been exposed to diverse visual patterns through training large-scale datasets such as ImageNet. This teacher encapsulates rich semantic knowledge that we transfer to guide our student decoder, providing reference point for anomaly detection through consistency enforcement. We apply this distillation mechanism specifically to query images.
Formally, given a query image, let क and Eiry denote feature maps from the i-th layer of the teacher encoder and student decoder, respectively. For knowledge transfer, we define the following loss function:
Lisd Σ HHWW 1imsi w), 2gry w|),
where (h, w) indexes all spatial locations in feature maps wyyy and z H4, and W; represent the height and width of the feature maps at the i-th layer, and sim(a, b) = a b// b/||a|| [[b]].
After training exclusively on normal data, the teacher encoder and student decoder become aligned to represent normal patterns with high similarity. For normal test query samples that conform to patterns observed during training, the student's representations closely match the teacher's, resulting in low distillation loss. Conversely, when presented with anomalous query samples, the student decoder fails to adequately reconstruct the teacher's feature maps, producing discrepancies in their representations.
Teacher-student distillation provides solid foundation but is incomplete for few-shot anomaly detection since it does not make use of support images. Given limited normal references available in few-shot settings, we introduce student self-distillation to capitalize on these valuable normal examples. By performing self-distillation on support images, we establish a consistent reference base of what "normal" looks like, against which query images can be compared. This design aligns with clinical diagnostic practices, where distinguishing pathological abnormalities from normal anatomical variations requires reliable reference data.
Student Self- Seff-Distillation
3

---

Published as a conference paper at ICLR 2026
ki Formally, given support set, let Sapt denote feature maps of the k-th support image from the i-th layer of the student encoder. We define a student self dist-iisiilationn loss that aggregates multi-scale discrepancies
Leod = K ΣΣ Σ 11
sim(=5ge (h, w], zgrg (h, w))
(2)
H.W,
h
where K represents the number of support images.
By minimizing this loss, the student network learns align the query's representations with normal references, which is crucial for few-shot anomaly detection. This self-distillation mechanism enables model to learn more discriminative features, even with the constraint of having access to only normal training samples.
Training Objective In summary, we train our model by optimizing a combination of the two distillation losses:
L XLred + Lead
(3)
where À a hyperparameter that balances the teacher-student distillation and student self-distillation to the overall learning objective.
2.3 LEARN-TO-WEIGHIT
In Eq. we initially assign equal weights to all support images. However, we observe that this assumption is often suboptimal the reference value provided by each support image varies for different query images. This phenomenon is particularly prevalent in medical imaging, where individual anatomical variability leads to significant differences in shape and appearance among support images, resulting in varied reference values for the same query.
To better utilize the support set, we propose tearn-to-weight mechanism that adaptively weights support samples based on their relevance to given query. Specifically, we learn reference values of support images conditioned on the query as follows:
w; = softmax Eory x db ((2apt))
VC:
where ø is linear projection implemented as 1x1 convolution. Note that appropriate reshaping oper-
ations are performed: zary x is reshaped to R H,, and x'pp E R xCx H, is reshaped to R (C, (x,
We then reformulate the student self delf-distillation loss by incorporating our farn-to-weight mechanism as:
Eary x 65
Land 12a
Σ
Σ
sim(softmax
) (h, w/, 2 [h, w)) (5)
H.W. h,w
VC.
The overall training objective is updated to:
XLead Lead
Our learn-to-weight mechanism enables more effective knowledge transfer from support to query images by dynamically assessing the importance of each support sample based on its relevance to the specific query being evaluation.
2.4 ANOMALY SCORING
At inference time, given test query image and K support images of novel task (representing unseen imaging modality and/or disease), we compute a similarity map at each scale using Eq. (5). These multi-scale similarity maps then aggregated to derive comprehensive anomaly map. The final anomaly score for the input image is obtained by computing the mean value of this aggregated anomaly map, providing scalar quantification of anomalousness.
4

---

Published as a conference paper at ICLR 2026
3 RELATED WORK
Given the expense and time required to obtain verified anomalies in medical imaging, unsuper- vised anomaly detection methods have become mainstream, Reconstruction-be based approaches have emerged as a leading paradigm in this field. |Schlegl et al. (2017) pioneer the use of GANs with AnoGAN (Li et al. 2018), later introducing f-AnoGAN (Schlegl et al.) 2019), which improves effi- ciency by incorporating encoder to map images to a latent space. Various autoencoder architectures have been explored, including variational autocncoders (Zimmerer et al.) 2018) and vector-quantized variational autoencoders (Naval Marimont and Tarroni [021)p To address the overgeneralization problem, where abnormal images are reconstructed too accurately, (iong et al. (2019) and Park et al. (2020) introduce memory banks store normal patterns for comparison during inference. Several works (Rudolph et al. [202]] Gudovskiy et al., 2022, Yu et al. (202I) leverage normalizing flows, enabling exact likelihood estimation for image modeling, and achieve strong performance anomaly detection.
Self-supervised learning (Jing and Tian 2021) has also been applied to anomaly detection, typically following two paradigms. One-stage approaches train models to detect synthetic anomalies and directly apply them to real abnormalities (Tan et al., 202T Schlüter et al. 2022). Two-stage ap- proaches first learn self- superviedd representations on normal data, followed by constructing one-class classifiers (Li et al., 2021]] [Sohn et al., 2021). Recently, knowledge distillation from pre-trained
models presents another promising approach for unsupervised anomaly detection (Salehi et al.) 202T Deng and Li, 2022; Batzner et al. 2024). In these methods, a student network distilled by pre-trained teacher network normal samples only extract normal features, leading to detectable discrepancies when anomalies are encountered during inference.
However, these methods typically require numerous normal samples for effective training- a require- ment that becomes impractical in scenarios with limited data availability. This limitation motivates the need for more efficient anomaly detection methods that achieve strong performance with few training samples.
Few-shot anomaly detection aims generalize novel tasks using limited reference samples. Early approaches such as TDG (Sheynin et al.) 2021) and DiffNet (Rudolph et al.] (021] primarily focus mne-model-per-tarn
on training with small set of normal samples, adhering necessitates retraining for each new anomaly detection task. RegAD (Huang et al., (2022) addresses this limitation by leveraging registration as a task-agnostic proxy, enabling generalization across unseen tasks. It is noteworthy that these methods are primarily designed for industrial defects and fail to account for the high variability present in medical imaging, where data span diverse modalities (e.g.. MRI and CT) and anatomical regions. In addition, their reliance on aggressive data augmentation (e.g., random rotation) is impractical for medical images with strict structural constraints.
paradigm that
In medical imaging, recent efforts like MediCLIP (Zhang et al., 2024) improve performance by synthesizing anomalies from limited normal data but remain confined to single-task scenarios. MVFA (Huang et al.) (2224) introduces multi-level visual feature adapters to align CLIP's visual and language representations, yet demands pixel-level annotations and anomalous samples during training, which costly to acquire in many scenarios. InCTRL (Zhu and Pangl (024) achieves cross-domain generalization via contextual residual learning but still requires anomalous data for optimization. INP-Former (Luo et al., 2025) achieves reference-free anomaly detection by extracting intrinsic normal prototypes from test images. However, this approach is designed for natural images and shows limited performance on medical imaging data. In contrast, our method trained solely on normal samples and, during inference, adapts to unseen anomaly detection tasks using only a few normal reference images, eliminating the need for annotated anomalies or retraining. This approach reduces deployment costs while maintaining strong generalization across medical modalities and anatomical regions.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets To evaluate our method, we compile a comprehensive dataset spanning diverse anatomical regions, lesion types, and imaging modalities by integrating several widely used medical anomaly
5

---

Published as a conference paper at ICLR 2026
Table I: Anomaly detection performance comparison on multiple medical imaging datasets (HIS, LAG, APTOS, RSNA, and Brain Tumor). Performance is measured using image-level AUROC (%) with the pest-performing method highlighted in bold. We compare our approach against both unsupervised and few-shot (4) methods.
Method
2 sho
HIS 4 shot
8 sbot
shot
TAG
shot
8 shot
Abot
APTOS shoe
8 shot
2 sbot
RSNA 4 shox
8 shot
2 shot
Brain Tumor
shot
sbot
44.8
48.5 2ua
2m
60.5
218
68.5
430
42.0g5a
68.0
664
2.10
1AAsGIAN
67.4123
58.0
km
31.3
30.5
122
43.1
22.2.26
34.3 155
53.9
FautPlw
73.1
21
66.
66.
66.522
22.2.15
878
744
77.9pm
70.6-00
908
90:3+11
91
PatchCore
67.8
54.7.ms
89
Laus
27.5
27.3
19.1
719
650
20.9
76.
66.12:9
76.0az
78 Lar
SimpleNet
00.5.43
64.5-11
55.1-50
70.3.13
09.3.ss
74.9.1
728-48
73.4119
73.2
62.-20
09:7-13
76.9-13
6355
70.5.1
77.4-56
CuuPase
57.9
24.0-01
73.-)
502222
77.2a04
78.1218
513
59.0g513
25.
67.3
74.
k3.3
79.3
87.7204
19.3
NSA
34.8
51.1
508-79
57.1 51.0226
48.
5-5.
39.9274
46.6
52.6-19
588-17
51.74:31
48.6
90.92147
18.956.
44 44.1
ReContrast
58.52m0
65.
65.2.m1
61.4223
72.7
77.5an2
52.6.55
75.3
75
62.4
71.1
75.5am
75.3
56.2218
63: 63.1a3m
71.4zo
RD
40.3-15
418-55
4277.a
41.9.ss
43.5
54.6.68
47.0
52.4-17
59.1-51
60.0..
58.5.a9
260
25.5.33
24.4.34
RD4AD
T3.2212
18. 68 21.
65.722
71.721. 77.0
51.3244
51
68. 68.7
67:4-2
60.7111
68.2214
68
76
62.2230
74.611
81 8144
SCRDSAD
68.611.
69:5.10
69.3.12
18.1.1
67.1
68.8.19
82.9.67
8366.11
67.
67.9114
67.3
81.8 81.4.L.
RepAD
54.5215
57.0-21
414422
51.9233
54.421.3
28.1
550g24
59.42223
21
461224
50.9211
61.522
83
70
2022am
758
WICCLIP
58.0ins
58.8
57.1
m5.0
53.8
55.0.pm
33.1.an
53.1
53.0 00
696 :00
70.8in
71.8
38.2
i10
45.0
43.4.a0
MaIiCLIP
61/4113
66.8217
71.7
72.5 73.2234
77.6.17
77.7
78.5.L1
50.721
59,4-43
899
88:2+12
88. 88.7
MD
76.42z
806:10
80.222
12.1
77.2sz: 81.0.3x
86.
$7.4254
89.6-11
746.58
57
57.4
81.9.1.
928
93:7234 26.
LOCTRL
TLM00: 73.5.a0
m27
T1.7.00
71.1.05 71.1.00
95.6.40
94.500
89.5-00
79.5.00
S1.4.0
K2.7.00
906-10
91.8.c0
918..00
INP Fonner
63.0g25
66.011
658_11
71.826
71.8213 74.7212
896
90.4224
91. 91.
26.
79.122
78.5-25
74.7
78A450
Ancensly GPT
47.9
47.1205
48.32)4
57.6
18.1
57.5210
762-17
78.7
--
628-34
63.021
63
62.7
769-11
78.934
78.9227
Ours
94.222
942 94.3
947.22
962.1.1 97.3ax
100.011
88.9mm
97.9
99.
95.
95.31 95.5pts
detection benchmarks. We provide detailed information on dataset composition and train/test splits below ensure transparency and reproducibility.
HIS: Derived from Camelyon 16 which contains 6,091 normal and 997 abnormal patches from breast cancer patients, we allocate 700 normal images for training and create balanced test set comprising 400 normal and 400 abnormal images.
LAG: This dataset (Li et al. 2019) consists of 6,882 normal retinal fundus images and 4,878 abnormal images exhibiting glaucoma. Following Cai et al. (2022), we utilize 1,500 normal images for training and evaluate on 811 normal and 911 abnormal images.
APTOS: This collectior] of retinal images from diabetic retinopathy patients provides 1,000 normal samples for our training set, while the test set comprises 805 normal samples and 1,857 anomalous samples.
RSNA: This chest X-ray datase accordance with Cai et al. (2022). we select 1,000 normal images for training and construct a balanced test set with 1,000 normal and 1,000 abnormal images.
contains 8,851 normal and 6,012 lung opacity images. In
Brain Tumor: This datase encompasses 2,000 MRI slices without tumors, 1,621 with gliomas, and 1,645 with meningiomas. We classify both glioma and meningioma slices anomalies. The normal cases originate from Br35H5 and /aalh et al. (2020), while the anomalous cases come from (Saleh et al. (2020) and Cheng et (201)). Following CCa et al.] (2022), our experimental protocol uses I, 1,000 normal slices for training and evaluates on test set containing 600 normal and 600 abnormal slices (equally distributed between glioma and meningioma).
These datasets present significant challenges, including subtle pathological manifestations, heteroge- neous lesion morphologies. and complex tissue architectures, thereby providing rigorous testbed for evaluating our model's 's generalization capabilities across multiple medical imaging scenarios.
Implementation Details We employ WideResNet-500 ZZaoruyko and Komodakis 2016) pre- trained on ImageNet as the backbone for our teacher encoder. Parameters are optimized using the Adam optimizer with 3 0.5 and (3 0.999 (Kingma and Bal 2014). Our model is trained for 70 epochs with a batch size of 64, and all images are resized to 128 x 28 pixels. All experiments are
conducted on single NVIDIA 1100 GPU.

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study results across five medical imaging datasets (HIS, LAG, APTOS, RSNA and Brain Tumor). We examine the impact of student self-distillattin (ssd), teacher-student distillation (tsd), and learn-to- te-height mechanism (12w). Performance is reported image-level AUROC (%) with the best configuration highlighted in bold.
H
LAG
APTOS 2-shot 4-shot
RSNA
Brain Tumor
and ted 12v
2-shot
4-shot
8-shot
2-shot
4-shot
8-sbot
8-shot
2-shot
4-shot
8-shot
2-shot
4-shot 8-shot
66.1
66.7
74.1
70.2
75.9
60.8
66.5
74.1
55.5
73.8
90.0
91.0
92.9
92.7
93.9
94.0
92.7
98.8
99.2
84.9
92.7
98.1
93.9
93.6
94.2
V
V
94.3
94.5
95.8
95.8
96.5
97.0
100.0
100.0
100.0
98.7
98.7
99.5
94.9
95.3
96.4
92.6
93.6
94.3
93.1
93.8
98.6
99.5
99.8
99.9
88.1
96.8
98.4
94.0
94.2
94.7
V
96.7
98.4
99.0
97.9
98.0
100.0
100.0
100.0
99.0
99.1
95.1
96.3
Competing Methods Following prior works (Jeong et al. [0233 Huang al. 2024 Gu 2024
we compare our approach against both unsupervised and few-shot methods. We include a broad set of tfateoo--the-ar art methods, including the unsupervised approaches (UAE (Mao al. 22020). f-AnoGAN
(Schlegl et al. 2019). FastFlow (Yu et al. (02T), PatchCore (Roth et al.) (022). SimpleNer (Liu
al. 2023), CutPaste (Li et al. 2021). 1), NSA (Schlüter et al. 2022). ReContrast (Guo al. 2023),
RD++ (Tien et al., 2023). RD4AD (Deng and Lij 2022) and SCRI4AD et al. (025) and the
few-shot methods (RegAD (Huang et al. 2022). WinCLIP (Jeong et al. 2023), MediCLIP (Zhang
ct al. ) MVFA (Huang al. 2024). InCTRL (Zhu and Pang 2024), INP-Former (Luo et al.
2025 and AnomalyGPT (Gu et al. 2024).. For unsupervised methods and few-shot methods without utilize K support (normal) images as training data to train the models and evaluate them on the same task. For few-shot methods with generalization capability (RegAD, WinCLIP, InCTRL, AnomalyGPT, and ours), we adopt leave-one-out training and testing protocol, where networks are trained on all datasets except the one being used for testing. Note that some baselines, such as PatchCore and RD4AD, compute image-level anomaly scores by applying max max-pooling over the pixel-level anomaly maps.
generalization capability (MediCLIP. MVFA, and INP-Formes
Evaluation Metrics Following prior works (Cao et al. 2023
Pang et al. 2023 Zhu and Pang
2024), we focus on image-level tasks, as many medical datasets do not provide pixel-level annotations. We use AUROC to quantify model performance, which is the standard metric for anomaly detection tasks.
4.2 RESULTS AND DISCUSSION
We present our experimental results Table 1 Across all evaluated settings (K = 2, 4, 8), our proposed method consistently outperforms competing approaches the HIS, LAG, APTOS, and RSNA datasets, demonstrating significant improvements over the second-best methods. On the Brain Tumor dataset, our approach achieves performance comparable MVFA. Unsupervised anomaly detection methods generally exhibit limited performance, likely due to insufficient training samples. In contrast, few-shot anomaly detection algorithms demonstrate superior performance, with our proposed approach showing particularly strong generalization abilities across diverse medical imaging datasets. With only two normal reference images (K 2), our method achieves 100% precision on the APTOS dataset. Note that results for MediCLII and InCTR differ from those reported in their original papers due to evaluation on different datasets.
For qualitative analysis, we demonstrate our method's anomaly localization capability through example anomaly maps in Appendix
In addition, we find that our model is not sensitive to the choice of support images (see Appendix C
4.3 ABLATION STUDIES
To evaluate each component's contribution to our approach, conduct comprehensive ablation studies across five datasets, with results presented in Table
Student distiiistillttiin plays a critical role in our few-shot anomaly detection framework. Without it, our network effectively degenerates into unsupervised approach.
Student Self-Distillation
7

---

Published as a conference paper at ICLR 2026
HIS
LAG
APTOS
RSNA
Brain Tumor
Normal
Anomaly
Figure 2: Distribution of abnormality scores for normal (blue) and abnormal (red) samples across five datasets, visualized using raincloud plots and boxplots. For each dataset, the left subplot presents results without the learn-to-weight mechanism, while the right subplot shows results with the mecha- nism applied. Reduced overlap between red and blue distributions indicates superior discrimination performance. Each boxplot displays the median value and interquartile range (IQR), with whiskers extending to the extrema within 1,5x IQR from the quartiles, illustrating the significance of our approach.
Table Analysis of weight coefficient À and its impact on anomaly detection performance.
HIS
LAG
APTOS
RSNA
Brain Tumor
2-shot
4-shot
8-shot
2-shot
2-shot
4-shot
2-shot
4-shot
4-shot
1.0
95.3
96.0
96.4
94.9
96.7
97.9
100.0
100.0
100.0
96.7
98.7
99.5
94.7
95.6
95.9
96.0
97.4
100.0
100.0
93.0
99.3
94.9
96.0
96.7
98.4
99.0
97.6
97.9
98.0
100.0
100.0
100.0
99.0
99.1
99.8
95.1
96.3
96.8
0.05
95.8
97.5
97.6
95.1
95.9
97.3
100.0
100.0
100.0
86.7
97.0
99.4
94.7
93.6
95.6
0.01
95.2
96.9
97.4
92.7
95.8
99.2
99.4
100.0
100.0
90.1
97.4
99.5
94.2
94.5
95.3
To rigorously validate its importance, we conduct an ablation study (see Table first row). The results demonstrate a significant performance degradation when student self-distillation is removed, confirming its efficacy and necessity in the few-shot setting.
Teacher-Student Distillation Our results demonstrate that teacher-student distillation substantially enhances model performance. Under various few-shot settings (K = 2, 4, 8), AUROC scores across the five datasets improve by margins ranging from 0.66% to 13.76% (cf. Table P We include qualitative t-SNE visualization in Appendix D to further examine the underlying mechanism.
Learn-to- Weight Mechanism Experimental results confirm the effectiveness of the proposed learn- to te-weight mechanism, as evidenced in Figure [ Without this mechanism. histograms of normal and abnormal samples exhibit significant overlaps. As detailed Table incorporating this component yields average AUROC improvement of 1.91% across the five datasets for K 2, 4, 8, with peak improvement reaching 6.81% %. Moreover, this module is domain-agnostic and can be integrated into various few-shot learning frameworks.
Weight Coeflicient The weight coefficient À balances the contribution between teacher-student distillation and student self-distillation in the overall objective function. To determine its optimal value, we conducted grid search over À E {1.0, 0.5, 0.1, 0.05, 0.01) using dedicated validation set containing 100 normal and 100 anomalous images per task. Results in Table 3 indicate that setting 0.1 yields the best average performance across all datasets on the test set.
4.4 IMPACT OF DIFFERENT BACKBONES
Table 4 presents quantitative comparison of different backbones serving as the teacher network. As
expected, deeper and wider networks demonstrate superior representational capacity, contributing to more accurate anomaly detection. Notably, our dual distillation approach maintains competitive performance even when employing more compact architectures (e.g.. ResNet-34 (Koonce) 2021)
Interestingly, observe that extremely large models such Swin Transformer (Liu et al. 202
8

---

Published as a conference paper at ICLR 2026
Table 4: Quantitative performance comparison of various backhone architectures employed the
teacher network across all datasets.
HIS
LAG
APTOS
RSNA
Brain Tumor
Backbone
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot
ResNet-34
87.0
90.3
92.0
92.8
95.9
96.1
100.0
100.0
100.0
92.1
93.9
95.3
89.2
93.0
94.9
ResNet-50
91.4
93.9
95.4
92.5
96.3
96.4
100.0
100.0
100.0
97.1
98.3
95.3
96.4
Swin-S
74.8
74,9
82.1
75.5
82.7
90.7
99.9
100.0
100.0
67.8
68.2
69.2
89.1
93.1
96.4
Swin-B
64.7
67.9
80.1
78.8
76.6
70.2
98.4
98.3
99.3
61.6
62.6
63.2
82.6
90.4
90.9
CLIP
88.5
91.2
93.5
93.1
93.8
97.8
99.9
99.9
85.7
90.4
93.1
92.8
93.5
95.2
BiomedCLIP
89.2
92.1
94.3
95.2
95.7
100.0
94.3
96.2
94.6
98.1
WideResNet-500
96.7
98.4
99.0
97.6
97.9
98.0
100.0
100.0
100.0
99.0
99.1
99.8
95,1
96.3
96.8
yield suboptimal results despite their theoretical capacity advantage. This counter-intuitive finding suggests that excessive complexity in the teacher network may impede effective knowledge transfer to the student network, creating a representational gap that the distillation process struggles to bridge. The phenomenon highlights an important trade-off in our teacher-student architecture where the ideal teacher model balances representational power with distillation compatibility.
4.5 INFERENCE TIME AND PARAMETERS
We compare our model with competing few-shot anomaly detection methods and a representative unsupervised method (RD4AD) in terms of AU- ROC, inference time, and memory usage during inference, averaged across all five datasets (see
D'4FAD
AUROC (%6)
80-
|aCTRI
Figure 3). All evaluations are conducted on NVIDIA RTX 3090 GPU (24 GB VRAM). Our D24FAD achieves the highest AUROC score anomaly detection while demonstrating sig- nificant computational
MVFA
70-
MediCLIP
AmomalyGPT
RD4AD
60
runningg 2 faster than MVFA, 2.5x faster than InCTRL and RegAD, and 3x faster than AnomalyGPT and WinCLIP. In addition, D24FAD requires only 5 GB of memory for inference, position- ing it among the most memory-efficient few- shot anomaly detection methods. These results demonstrate the superior practical value of our approach compared to competitors.
50
RegAD
10
20
70
80
Inference Time (ms)
Figure 3: Performance comparison of anomaly de- tection methods across three dimensions: AUROC score (vertical axis), inference time (horizontal axis), and memory footprint (circle radius).
CONCLUSION
In this paper, introduce D²4FAD, novel dual distillation framework for few-shot anomaly detection in medical imaging. By leveraging pre-trained encoder as a teacher network and employing student decoder that distills knowledge from the teacher on query images while self-distilling on support images, our approach effectively identifies anomalies in novel tasks using only a small set of normal reference images. The by dynamically assessing the reference value of each support image conditioned on the query.
mechanism we proposed further enhances performance
Our extensive experiments on a comprehensive benchmark dataset comprising 13.084 images across multiple organs, imaging modalities, and disease categories demonstrate that D24FAD significantly outperforms existing methods in few-shot medical anomaly detection. Specifically, our approach achieves superior AUROC scores while maintaining computational efficiency. Furthermore, 424D exhibits remarkable memory efficiency, requiring only GB for inference, which positions it among the most efffccieeeeffcientt few-shot anomaly detection methods.
The clinical significance our work is particularly notable medical settings where obtaining large annotated datasets is challenging due to privacy concerns, rare pathologies, resource constraints. By requiring only few normal samples, D'4 enables practical anomaly detection across diverse medical imaging scenarios without extensive data collection efforts. This makes our approach

---

Published as a conference paper at ICLR 2026
especially valuable for detecting anomalies in uncommon anatomical structures rare disease presentations where comprehensive datasets are unavailable.
ACKNOWLEDGMENTS
This work supported in part by the Shaanxi Province Young Talent Support Program and the Qin Chuangyuan Innovation and Entrepreneurship Talent Project (H023360002), and part by the China Postdoctoral Science Foundation (Nos. 2023M742737 and 2023TQ0257).
ETHICS STATEMENT
The authors acknowledge that this work adheres to the ICLR Code of Ethics.
REFERENCES
Kilian Batzner, Lars Heckler, and Rebecca König. EfficientAD: Accurate visual anomaly detection millisecond-level latencies. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 128 138, 2024.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR), volume pages 60-65, 2005.
Yu Cai, Hao Chen, Xin Yang, Yu Zhou, and Kwang- -ing Cheng. Dual- diul-distribution discrepancy for anomaly detection in chest X-rays. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention (MICCAI), pages 584-593, 2022.
Yu Cai, Weiwen Zhang, Hao Chen, and Kwang-Ting Ting Cheng. Medianomaly: comparative study of anomaly detection medical images. Medical Image Analysis, page 103500, 2025.
Tri Cao, Jiawen Zhu, and Guansong Pang. Anomaly detection under distribution shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6511 -6523, 2023.
Jun Cheng, Wei Huang. Shuangliang Cao, Ru Yang, Wei Yang. Zhaoqiang Yun, Zhijian Wang, and Qianjin Feng. Enhanced performance of brain tumor classification via tumor region augmentation and partition. Public Library of Science One, (0110)e014038) 2015.
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9737-9746, 2022.
Tharindu Fernando, Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Deep learning for medical anomaly detection-A survey. ACM Computing Surveys (CSUR), 54(7): 1-37, 2021.
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1705-1714, 2019.
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. AnomalyGPT. Detecting industrial anomalies using large vision-language models. In Proceedings the AAAI Conference on Artificial Intelligence (AAAI), pages
2024.
Zhihao Gu, Liang Liu, Xu Chen, Ran Yi, Jiangning Zhang. Yabiao Wang. Chengjie Wang, Annan Shu, Guannan Jiang, and Lizhuang Ma. Remembering normality: Memory-guides knowledge distillation for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 16401-16409, 2023.
10

---

Published as a conference paper at ICLR 2026
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. CFLOW-AD: Real- time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Viston (WACV), pages 98-107, 2022.
Jia Guo, Shuai Lu, Lize Jia, Weihang Zhang, and Huiqi Li. ReContrast: Domain speciffc anomaly detection via contrastive reconstruction. In Proceedings of the International Conference Neural Information Processing Systems (NeurIPS), pages 10721-10740, 2023.
Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang. Registration based few-shot anomaly detection. In European Conference on Computer Vision (ECCV), pages 303-319, 2022.
Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang. Xinchao Wang, and Yanfeng Wang. Adapting visual- language models for generalizable anomaly detection in medical images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11375- 11385, 2024.
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. WinCLIP: Zero-ffew-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19606-19616, 2023.
Wenqian Jiang, Yang Hong, Beitong Zhou, Xin He, and Cheng Cheng. A GAN-based anomaly detection approach for imbalanced industrial time series. IEEE Access, 7:143608-143619 2019.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1 (1:11))437705555 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Brett Koonce. Resnet 34. In Convolutional Neural Networks with Swift for TensorFlow: Image Recognition and Dataset Categorization, pages 51-61. 2021.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. CutPaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9664 9674, 2021.
Chunlei Li, Yilei Shi, Jingliang Hu, Xiao Xiang Zhu, and Lichao Mou. Scale-Aware contrastive reverse distillation for unsupervised medical anomaly detection. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. Poster presentation.
Dan Li, Dacheng Chen, Jonathan Goh, and See-Kiong Ng. Anomaly detection with generative 2018
adversarial networks for multivariate time series. arXiv preprint
Liu Li, Mai Xu, Xiaofei Wang, Lai Jiang, and Hanruo Liu. Attention based glaucoma detection: A large-scale database and CNN model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages
2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 10012-10022, 2021.
Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. SimpleNet: A simple network for image anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2023.
Wei Luo, Yunkang Cao, Haiming Yao, Xiaotian Zhang, Jianan Lou, Yuqi Cheng, Weiming Shen, and Wenyong Yu. Exploring intrinsic nonmal prototypes within single image for universal anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9974 9983, 2025.

---

Published as a conference paper at ICLR 2026
Ye Ma, Xu Jiang, Nan Guan, and Wang Yi. Anomaly detection based multi-teacher knowledge distillation. Journal of systems architecture, 138: 1028610 2023.
Yifan Mao, Fei-Fei Xue, Ruixuan Wang, Jianguo Zhang, Wei-Shi Zheng, and Hongmei Liu, Abnor- mality detection in chest X-ray images using uncertainty prediction autoencoders. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 529-538, 2020.
Xueyan Mei, Zelong Liu, Philip M Robson, Brett Marinelli, Mingqian Huang, Amish Doshi, Adam Jacobi, Chendi Cao, Katherine E Link, Thomas Yang. et al. Radimagenet: An open radiologic deep learning research dataset for effective transfer learning. Radiology: Artificial Intelligence, (5)=210035, 2022.
Sergio Naval Marimont and Giacomo Tarroni. Anomaly detection through latent space restoration using vector quantized variational autoencoders. In IEEE International Symposium on Biomedical Imaging (ISBI), pages 1764-1767, 2021.
Eva Pachetti and Sara Colantonio. A systematic review of few-shot learning in medical imaging. Artificial Intelligence in Medicine, 156: 102949, 2024.
Guansong Pang, Chunhua Shen, Huidong Jin, and Anton Van Den Hengel. Deep weakly-supervised anomaly detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 1795-1807, 2023.
Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided normality for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 14372-14333 14381, 2020.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard SchöIkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14318 14318-14328, 2022.
Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semi-supervine defect detection with normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1907-1916, 2021.
Ahmad Salch, Rozana Sukaik, and Samy Abu-Naser. Brain tumor classification using deep learning. In International Conference on Assistive and Rehabilitation Technologies (CCareTech) pages 131-136, 2020.
Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H. Rohban, and Hamid R. Rabiee. Multiresolution knowledge distillation for anomaly detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 14902-14912. 2021.
Thomas Schlegl, Philipp Seebück, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Information Processing in Medical Imaging (IPMI), page 146-157, 2017.
Thomas Schlegl, Philipp Seebück, Sebastian M. Waldstein, Georg Langs, and Ursula Schmidt-Erfurth. f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis, 54:30 44, 2019.
Hannah M. Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz. Natural synthetic anomalies for self-supervised anomaly detection and localization. In European Conference on Computer Vision (ECCV), pages 489, 2022.
Shelly Sheynin. Sagie Benaim, and Lior Wolf. A hierarchical transformation- crative model for few shot anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8495-8504, 2021.
gen-
Nina Shvetsova, Bart Bakker, Irina Fedulova, Heinrich Schulz, and Dmitry V. Dylov. Anomaly detection in medical imaging with deep perceptual autoencoders. IEEE Access, 9:118571-118583, 2021.
12

---

Published as a conference paper at ICLR 2026
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluat- ing representations for deep one-class classification. In International Conference on Learning Representations (ICLR), 2021.
Jeremy Tan, Benjamin Hou, Thomas G. Day, John M. Simpson, Daniel Rueckert, and Bernhard Kainz. Detecting outliers with poisson image interpolation. In International Conference on Medical Image Computing and Computer-Assiste Intervention (MICCAI), pages 581-591, 2021.
Tran Dinh Tien, Anh Tuan Nguyen, Nguyen Hoang Tran, Ta Duc Huy, Soan Duong, Chanh D. Tr. Nguyen, and Steven Q. H. Truong. Revisiting reverse distillation for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24511 2-5220,,,,, 2023.
Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. FastFlow: Unsupervised anomaly detection and localization via 2D normalizing flows. arXiv preprint arXiv:2111.07677, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 2016.
Ximiao Zhang, Min Xu, Dehui Qiu, Ruixin Yan, Ning Lang, and Xiuzhuang Zhou MediCLIP: Adapting CLIP for few-shot medical image anomaly detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 458 468, 2024.
Qihang Zhou, Shibo He, Haoyu Liu, Tao Chen, and Jiming Chen. Pull push: Leveraging differen- tial knowledge distillation for efficient unsupervised anomaly detection and localization. IEEE Transactions on Circuits and Systems for Video Technology, 331))2217--218,, 2022.
Jiawen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learning with few-shot sample prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17826-17836, 2024.
David Zimmerer, Simon A. A. Kohl, Jens Petersen, Fabian Isensee, and Klaus H. Maier-Hein. Context-encoding variational autoencoder for unsupervised anomaly detection. arXiv preprint arXiv:1812.055941, 2018.
APPENDIX
A USE OF LARGE LANGUAGE MODELS
Large language models were used solely for light editing tasks including grammar correction, spelling checks, and minor phrasing improvements to enhance clarity and concision.
B ANALYSIS OF PRE-TRAINING DATA FOR THE TEACHER NETWORK
To examine how the teacher's pre-training domain influences performance, we compare ResNet-50 models trained on ImageNet (natural images) and RadImageNet (Mei et al. 2022) (medical images). As shown in Table the medically pre-trained teacher provides an average improvement of 0.91 % indicating that domain-aligned representations enhance accuracy. This also reinforces our main claim that the proposed distillation framework is flexible and can readily incorporate stronger or domain-specific teacher models.
C IMPACT OF SUPPORT IMAGE SELECTION
To evaluate the robustness of our method, we study the effect support image choice. We perform five trials, each with randomly selected support examples, and report the and standard deviation. As shown in Table performance with randomly selected support images differs only slightly from that with fixed set, indicating that support sample selection has minimal impact the results.
13

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison between different pre-training datasets ResNet-50 backbone.
HIS
LAG
APTOS
RSNA
Brain Tumor
2-shot
4-shot 8-shot
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot
2-shot
4-shot
8-shor
2-shot 4-shot 8-shot
ImageNet
91.4
93.9
95.4
92.5
96.3
96.4
100.0
100.0
100,0
94.2
97.1
98.3
94.6
95.3
96,4
RadImageNett
93.6
95.2
95.6
93.0
93.9
96.8
99.9
100.0
100.0
97.4
98.0
98.6
95.3
97.7
98.4
Table Performance comparison between fixed (top) and randomly sampled (bottom) support sets.
HIS 4-shot
LAG 4-shot
APTOS
RSNA
Brain Tumor
2-4hot
8-shot
2-thoe
S-hhot
2-4hot
4-shoe
8-thot
2-shot
4-sbot
8-thot
2-shot
4-thot
8-shot
D.
93.
93.5
93.2....
94.4.1:
96.1.1
27.
100.0.o..
100.0
100.0.102
88.4
97.3
98.9.n 95.8 95.3
95.2ams
92,422 934 93.5
20
99.8
100.0
93.4
96.0.1. 96.
D SEACHER-STUDERS DISTILLATION
We visualize feature representations from the student network using t-SNE the RSNA dataset. As illustrated in Figure using teacher student distillation produces more compact and discriminative feature clusters, validating its effectiveness.
Normal
Normal
Anomaly
Amomaly
20
10
0
-10
-20
-30
-30
-20
-20
Figure 4: t-SNE visualization of embeddings from normal and abnormal samples in the RSNA dataset, extracted from the student network. Left: embeddings without teacher teacher-student student distillation. Right: embeddings with teacher-student distillation applied.
E QUALITATIVE RESULTS
For qualitative analysis, we demonstrate our method's anomaly localization capability using anomaly maps shown in Figure 5 The figure displays results across five datasets: HIS, LAG, APTOS, RSNA, and Brain Tumor (from left to right). The top row shows original images from each dataset, while the bottom row shows the corresponding anomaly maps generated by our method. Despite promising qualitative localization, the absence of pixel-level annotations in our benchmark hinders quantitative evaluation, and this limitation could be addressed with fully annotated datasets in the future.
ALTERNATIVE INSTANTIATIONS OF THE LEARN-TO- TEARN-TO-WEIGHT MECHANISM
To assess the flexibility of our weighting mechanism, introduce three alternative similarity functions Gaussian, embedded Gaussian, and concatenation. to compute support similarity, followed by softmax normalization:
14

---

Published as a conference paper at ICLR 2026
Figure 5: Visualization exemplar anomaly maps generated by the proposed model.
Gaussian
W4, softmax " (Zige)
Fuclidean distance, as in (Buades et al. 2005), is also applicable, but the dot product is more implementation-friinn frrenndly in modern deep-learning frameworks.
Embedded Gaussian
softmax(e" (0
(8)
where and & are linear projections implemented with x 1 convolutions.
Concatenation
w; = (oottaa
(9)
where | denotes concatenation, and w is a learnable weight vector. The query feature zary R.ICix broadcast to (C. for this computation.
Table 7: Performance comparison between different ways to instantiate Learn- Searm-To-we Weight.
APTOS
RSNA
Brain Tumor
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot
2-shot
4-shot
8-shot 2-shot
4-shot 8-shot
2-shot 4-shot 8-shot
Guussian
95.7
97.4
98.0
95.6
97.0
99.9
99.2
97.0
98.8
95.3
95.8
Gaussian, embed
96.0
98.6
98.4
96.9
97.2
97.4
99.9
100.0
100.0
98.8
98.7
94.5
95.7
96.2
Concatenation
95.9
97.6
98.2
96.7
97.1
97.7
99.9
99.8
100.0
97.5
97.9
99.0
94.3
94.5
96.0
Ours
96.7
98.4
99.0
97.6
97.9
98.0
100.0
100.0
100.0
99.0
99.1
99.8
95.1
96.3
96.8
The remaining steps follow Section 2.3
As shown in Table 7 different weighting strategies produce comparable but distinct results. The scaled dot-product formulation used in our main model shows slightly better performance, likely because it provides more direct and effective measure of fea- ture similarity. These variants demonstrate the flexibility of our -tan-tt-weight design, and other alternatives may further improve performance.
We also visualize the learned weights in Figure Support images that are normal, anomalous, or semantically irrelevant marked in green, red, and yellow, respectively. Across multiple layers
15

---

Published as a conference paper at ICLR 2026
(weight 1/2/3), the model consistently assigns lower weights to anomalous irrelevant support images, highlighting the effectiveness of the proposed mechanism.
weight 0.5653
weight I 0.5824
weight 1 0.8092
weight 0.5324
weight 2; 0.6216
weight 2; 0.8225
weight
weight 3; 0.7095
weight 3: 0.8169
weight 0.4347
weight 1; 0.4176
weight 0.1908
weight 0.4676
weight 2, 0.3784
weight 2, 0.1775
Query
weight 0.4420
weight 0.2905
weight 0.1831
Figure 6: Visualization of learned weights in the learn-to- Sarn-to-weight weight mechanism.
16


Reviews:
[{"review_id": "8XJLCo7zzZ", "reviewer": "Reviewer_gRTX", "summary": "The paper proposes a dual-distillation framework for few-shot anomaly detection in medical images. The method freezes an ImageNet-pretrained encoder as a “teacher” and trains a lightweight decoder as a “student,” combining (1) feature distillation on query images and (2) self-distillation on support images. A 5-dataset benchmark (13,084 images) is introduced, and the method is shown to outperform prior work in image-level AUROC under 2/4/8-shot settings.", "questions": "- Clarify the evaluation setting for MediCLIP, MVFA, and INP-Former\n- Does the method support pixel-level localization? If yes, why are localization metrics not reported?\n- Why does Table 1 report lower performance than the last row of Table 2? \n- Have you experimented with stronger pretrained teachers (e.g., CLIP-ViT) instead of ImageNet CNNs?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear motivation and task definition for few-shot anomaly detection in medical settings.\n- Architecture is simple, fast, and avoids large generative models.\n- Strong image-level AUROC across multiple datasets and shot settings.", "weaknesses": "- The work repeatedly emphasizes “dual distillation” as a key contribution, but the process does not fully match established definitions of distillation in the literature. Since the teacher network is frozen, and the student is not learning logits or semantic knowledge but merely reconstructing features, the term distillation may be overstated. This weakens the conceptual positioning of the contribution: the method is an anomaly-detection reconstruction framework rather than a genuine knowledge-transfer framework. A more precise formulation would strengthen the paper’s technical clarity.\n- The fairness of the baseline comparison is still unclear. Although it is likely that MediCLIP, MVFA, and INP-Former are evaluated using their official pretrained checkpoints followed by few-shot inference (rather than re-trained from scratch), this introduces a different concern: the pretraining data used for these models may not be aligned with the data available to the proposed method. Since these baselines were originally trained on large external datasets (e.g., CLIP pretraining or full normal medical datasets), it is important to clarify whether (1) their pretraining data overlaps with the test datasets, and (2) whether the proposed method has access to comparable pretraining resources. Without such clarification, the performance difference may reflect differences in dataset exposure rather than architectural effectiveness.\n- Only image-level detection is evaluated, no localization metrics. Pixel-level anomaly localization (e.g., heatmaps, PRO, Dice, pixel-AUROC) is essential for clinical use. Several baselines do support localization (e.g., MVFA, AnomalyGPT), but the paper does not report or discuss localization performance, making the clinical impact claim incomplete.\n- Inconsistent reporting of “Ours” across tables. The AUROC results for the proposed method in Table 1 are lower than those in the final row of Table 2, even though both appear to evaluate the same benchmark. This suggests the two tables use different configurations (e.g., stronger backbone, more loss terms, or ablation-optimized settings), but this is not stated anywhere. It becomes unclear which version represents the “main result,” and whether the strongest performance was omitted from the primary comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RmN0XuCQwb", "reviewer": "Reviewer_fFwa", "summary": "This paper proposes $D^24FAD$, a dual-distillation framework for few-shot anomaly detection in medical imaging. The method employs a pre-trained encoder on ImageNet as a teacher and a learnable decoder as a student. The student distills knowledge from the teacher on query images and performs self-distillation on a few normal support images, creating a compact representation of “normal” anatomy for each task. A learn-to-weight module dynamically adjusts the contribution of each support sample conditioned on the query. The approach is trained only on normal data and evaluated in a few-shot setting (2-, 4-, and 8-shot) across five medical datasets spanning different modalities and organs. Experiments show consistently good AUROC results and competitive inference efficiency.", "questions": "1.Task formulation and localization:\nCan you provide quantitative results for anomaly localization (e.g., Dice, IoU) to demonstrate the spatial detection ability of your method? If such annotations are not available, could you at least show quantitative proxy metrics or additional qualitative evidence supporting the localization quality shown in Appendix B?\n\n2. Evaluation of segmentation-based baselines:\nSeveral baselines such as PatchCore and RD4AD were originally designed for pixel-level anomaly localization. How were these methods adapted to the image-level AUROC evaluation used in this paper? Could you clarify whether their localization outputs were averaged or thresholded, and how this might affect the reported results?\n\n3. Weight coefficient λ selection:\nHow was the optimal value of λ = 0.1 (Table 3) determined? Was there a validation set or a held-out meta-task used for tuning, or was the selection based on post-hoc test performance? A clear description of this procedure would help assess the fairness of comparisons.\n\n4. Inference time and memory usage (Figure 3):\nFor which dataset(s) were the inference time and memory measurements obtained? Are the reported values averaged across datasets or computed on a single representative dataset? \n\n5. Architectural flexibility:\nDoes the proposed framework require the teacher and student networks to share an identical architecture and layer resolution? If not, are there any projection or adaptation modules that enable distillation between heterogeneous backbones?\n\n6. Pretraining domain and robustness:\nHave you considered evaluating the method with a medical-domain pretrained teacher network (e.g.,  BioMedCLIP)? Such experiments would clarify whether the reliance on ImageNet pretraining affects the generalization ability across medical modalities.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a clear and well-motivated dual-distillation framework ($D^24FAD$) for few-shot anomaly detection, combining a teacher–student distillation mechanism with an additional student self-distillation path and a learn-to-weight module that adaptively re-weights support images conditioned on the query. While the core components (knowledge distillation, few-shot learning) are known, their integration into a unified few-shot medical anomaly detection framework is novel and conceptually elegant. The problem formulation, detecting unseen anomalies from only a few normal references, addresses a challenging and underexplored setting in medical imaging.\n\n2. The work is technically solid and empirically well supported. The authors evaluate their approach on five heterogeneous medical datasets and include ablation studies that clearly demonstrate the contribution of each component. The training procedure is reproducible and relies on standard architectures and optimization methods. The model achieves consistently strong performance, often surpassing baselines by a substantial margin, while maintaining efficient inference and low memory consumption. It should be noted, however, that several competing methods (e.g., PatchCore, RD4AD) were originally designed for pixel-level anomaly localization. In this paper they are evaluated only at the image level, which might underestimate their actual capability. Consequently, while the reported results indicate strong performance, the comparative advantage should be interpreted with some caution.\n\n3. The paper is clearly organized and readable. The motivation, methodology, and evaluation are logically connected.\n\n4. The paper targets an important real-world problem, automatic anomaly detection under limited supervision in medical imaging, which has substantial clinical implications.", "weaknesses": "The main limitation of the paper lies in the formulation of the task. Although the work is presented as addressing few-shot anomaly detection, the evaluation is restricted to image-level AUROC, effectively turning the problem into a binary classification task (normal versus abnormal). While the model internally produces anomaly maps, no quantitative localization results are provided (e.g., Dice, IoU, or AUPRO). This simplification reduces the methodological complexity of the problem and limits the demonstrated clinical applicability of the approach, where spatial localization of pathological regions is often essential.\n\nA second concern relates to the evaluation of competing methods. Several baselines included in the comparison, such as PatchCore and RD4AD, were originally designed for pixel-level anomaly localization. Converting these approaches to image-level evaluation may underestimate their true capability and potentially exaggerate the reported relative gains of the proposed model. A clearer justification or separate localization-based comparison would strengthen the empirical claims.\n\nIn terms of experimental details, several important aspects remain unspecified. The procedure used to select the weighting coefficient λ (Table 3) is not described; it is unclear whether this was tuned using a validation set or chosen post-hoc on test performance, which risks overfitting on test data. Similarly, the results reported in Figure 3 for inference time and memory usage are not linked to a particular dataset, making the comparison less transparent.\n\nThe framework’s architectural flexibility also appears limited. Because the distillation losses are computed layer-wise between teacher and student features of matching dimensions, both networks must share a compatible structure. The paper does not mention whether any projection or adaptation modules were used to support heterogeneous architectures.\n\nFinally, the analysis remains confined to a teacher pretrained on ImageNet, which introduces a potential domain mismatch for medical images. The effect of employing a medical-domain pretrained backbone (e.g. BioMedCLIP) is not investigated. Such an analysis would provide valuable insight into the robustness and domain generalization of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQlw8CYamJ", "reviewer": "Reviewer_2kEb", "summary": "This paper introduces D²FAD, a novel framework for few-shot anomaly detection (FAD), with a specific focus on medical imaging. The core problem addressed is the scarcity of normal reference images in many clinical scenarios, which limits the applicability of traditional unsupervised anomaly detection methods. D²FAD employs a dual distillation strategy: 1) Teacher-Student Distillation, where a student decoder learns to reconstruct multi-scale features from a pre-trained (and frozen) teacher encoder on query images, and 2) Student Self-Distillation, where the student decoder learns to produce consistent representations between a query image and a small support set of normal images from the same domain. Additionally, the authors propose a \"learn-to-weight\" mechanism to dynamically assign importance to different support images based on their relevance to the query. The authors also contribute a new comprehensive benchmark dataset for medical FAD, curated from several public sources. Experiments show that D²FAD significantly outperforms a wide range of unsupervised and few-shot anomaly detection methods across this new benchmark.", "questions": "Regarding the \"Learn-to-Weight\" Mechanism: This is an interesting component. Could you provide some qualitative analysis to help understand what it learns? For instance, if you provide a \"bad\" or less relevant normal image in the support set, does the model learn to assign it a lower weight w_i? Visualizing these weights for a few examples could provide valuable insight.\n\nImpact of Teacher Model: Your results in Table 4 show that WideResNet-50 is the best teacher. This is consistent with many distillation-based anomaly detection papers. However, you also note that very large models like Swin Transformers perform suboptimally. Do you have an intuition for why this \"representational gap\" occurs? Is it because the student decoder is too simple to distill from such a complex teacher, or is there another reason?\n\n\nDataset Details: Thank you for curating this excellent benchmark. Could you clarify if the train/test splits for the leave-one-out protocol are strictly separated by dataset source (e.g., train on HIS, LAG, APTOS, RSNA; test on Brain Tumor) or by anatomy/modality? For instance, if you train on all datasets except Brain Tumor (MRI), would you also exclude other MRI datasets from the training set to test for modality generalization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear Motivation and Problem Formulation: The paper does an excellent job of motivating the need for few-shot anomaly detection in clinical practice, grounding the research in a real-world problem. The formalization of the FAD task is clear and precise.\n\nElegant and Effective Method: The D²FAD framework is simple yet powerful. The dual distillation concept is intuitive and well-justified. By using a frozen pre-trained encoder as the teacher, the method is parameter-efficient and avoids the need for complex architectural designs.\n\nThorough and Convincing Experiments: The empirical evaluation is a major strength. The authors have not only proposed a new method but have also built a strong benchmark to validate it. The comparisons against a wide array of baselines are comprehensive, and the ablation studies provide strong evidence for the efficacy of each component of their proposed method. The consistently high performance across multiple datasets and few-shot settings (2, 4, 8-shot) is very impressive.\n\nHigh Practical Relevance: The method is well-aligned with clinical reality, where often only a few trusted normal cases are available for reference. The high AUROC scores, combined with the method's efficiency (as shown in Figure 3), suggest strong potential for practical deployment.", "weaknesses": "Limited Technical Depth in \"Learn-to-Weight\": While the \"learn-to-weight\" mechanism (Eq. 4) is a good idea, its presentation is somewhat brief. It is essentially a scaled dot-product attention between the query and support features. The paper could benefit from a deeper analysis or discussion of this component. For example, are there other ways to instantiate this weighting? How does this mechanism behave in practice (e.g., does it learn to ignore outlier-like support images)?\n\nSensitivity to the Pre-trained Teacher: The entire framework's performance is heavily dependent on the quality of the frozen teacher encoder (pre-trained on ImageNet). While the ablation in Table 4 explores different backbones, it raises a question: how does the choice of pre-training dataset (not just architecture) affect performance? Would a teacher pre-trained on a large medical dataset (if available) perform even better? A brief discussion on the limitations imposed by the teacher's domain (natural images vs. medical) would strengthen the paper.\n\n\"Simplicity\" as a Double-Edged Sword: While the architectural simplicity is a strength, some might argue that the individual components (knowledge distillation, self-distillation, attention) are themselves not new. The novelty lies in their specific combination for the FAD task. This is a minor point, as the combination is non-trivial and highly effective, but it is worth noting that the paper's contribution is more about a novel framework than a fundamentally new algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dcquyKpwB4", "reviewer": "Reviewer_9qcV", "summary": "This paper focuses on image-level anomaly detection across anatomical contexts. It proposes a dual distill framework for anomaly detection in few-shot setting, where only up to 8 normal images as references. Specifically,  a spatial feature reconstruction loss is implemented for query images, and a spatially discrepancy loss between the query image and references images are implemented. The proposed framework is highly closed to the realistic medical setting, the proposed framework achieves satisfactory results.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n- This paper is well-written and easy to follow. \n- The proposed dual distillation framework is effective and easy to implement. \n- The experiments are extensive.", "weaknesses": "Weaknesses:\n -  The introduction part from Line 52- Line 54 can be better motivated.\n   - 1] actually leverages a large amount of unlabeled abnormal data for anomaly detection. I don’t quite understand why the authors cite this paper here, since they are arguing that, in some cases, normal data can be rarer than abnormal data.\n   -  The authors should have supported reference papers for the statement/claim from Line 54-55.\n- Figure 1 is not very informative. For example, the authors could include notations such as  X to represent the feature space of the teacher encoder and Z for the space of the student decoder. In addition, an illustration showing how the proposed loss terms are computed would make the figure more informative and easier to understand.\n - The title and the abstract are a bit mis-leading. After reading the paper, it seems to me that the inference is few-shot but not the training. My understanding is that only $K$ images for training if the setting is $K$-shot. However, the Line 291-309 suggest the training requiring the whole dataset. Please clarify this if I am wrong.\n\nMinor weaknesses: \n- Line 63 should be student decoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)