Title: Fence off Anomaly Interference: Cross-Domain Distillation for Fully Unsupervised Anomaly Detection
Venue: NeurIPS 2025
Accepted: no
Keywords: Anomaly Detection

Unsupervised Learning

Knowledge Distillation
Abstract: Fully Unsupervised Anomaly Detection (FUAD) is a practical extension of Unsupervised Anomaly Detection (UAD), aiming to detect anomalies without any labels even when the training set may contain anomalous samples. To achieve FUAD,  we pioneer the introduction of Knowledge Distillation (KD) paradigm based on teacher–student framework into the FUAD setting. However, due to the presence of anomalies in the training data, traditional KD methods risk enabling the student to learn the teacher’s representation of anomalies under FUAD setting, thereby resulting in poor anomaly detection performance. To address this issue, we propose a novel Cross-Domain Distillation (CDD) framework based on the widely studied reverse distillation (RD) paradigm. Specifically, we design a Domain-Specific Training, which divides the training set into multiple domains with lower anomaly ratios and train a domain-specific student for each. Cross-Domain Knowledge Aggregation is then performed, where pseudo-normal features generated by domain-specific students collaboratively guide a global student to learn generalized normal representations across all samples. Experimental results on noisy versions of the MVTec AD and VisA datasets demonstrate that our method achieves significant performance improvements over the baseline, validating its effectiveness under FUAD setting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a model for a setting with image data which they call fully unsupervised outlier detection, which comes with the assumption that a certain fraction of the training data is anomalous. \n\nThe model is based on a WideResNet-50 as an initial feature encoder (lines 119, 267). The extracted features are then processed by two sets of autoencoders (line 120, Figure 2, eq15 as loss for set 1, eq 20 as loss for set 2). \n\nOne set is trained over subsets of the training data, which have samples added to them which are called high confidence samples. The meaning of high confidence is a high cosine similarity between the original WideResNet-50 feature embedding and the output (= reconstructed features) of the second (\"global\") autoencoder from the previous epoch. \n\nThe second single (\"global\") autoencoder is trained apparently on the whole training data, but the features are selected from the outputs first set of autoencoders. Selection is based on similarity between the output of the second (\"global\") autoencoder from the previous epoch and the outputs from the set og autoencoders from the first set. It seemingly uses the selected sample as reconstruction target for the second (\"global\") autoencode.\n\nOverall the idea is that the autoencoders are run in an underfitting setting, and that for that reason they might have a worse reconstruction capability on samples in low density regions of the feature space relative to high density regions.\n\nAt inference time, this idea is employed by using eq (11) to predict outliers based on low similarity between the original WideResNet-50 feature embedding, and the output of the second autoencoder.\n\nThey measure performance on MVTec-AD dataset and VisA dataset. They compare it against 3 fully supervised baselines for one fixed outlier contamination ratio.\n\nThey evaluate varying number of subsets for the first set of autoencoders, and also different strategies how to select features from the first set as reconstruction target for the second autoencoder in Section 5.3\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- there is an incremental novelty by using a two tiers of autoencoders, consisting of the first set of autoencoders and the second global autoencoder, with the intention to strengthen the underfitting.\n\n- Experimental results are slightly better than the baselines, in particular on within image localization.\n\nWeaknesses:\n\n- one seemingly false claim about methodology:\n\n  lines 54: \"we propose a novel cross-domain distillation framework for FUAD,\n  built upon the widely studied KD-based UAD method Reverse Distillation (RD) [9]\"\n\n  They are not using Reverse Distillation according to paper [9]:\n\n  paper [9] https://openaccess.thecvf.com/content/CVPR2022/papers/Deng_Anomaly_Detection_via_Reverse_Distillation_From_One-Class_Embedding_CVPR_2022_paper.pdf \n  in Figure 2 defines reverse distillation as inputting a high-level feature and having a network reconstructed lower level features. \n\n    - as per lines 119-120 in this submission they input all L extracted layers into the network and train it to reconstruct exactly these L layers. \n\n  this is why this is considered a false claim about methodology. With this the whole section talking about reverse distillation in the paper is moot and hangs in the air. \n\n&nbsp;\n\n- The terms \"cross-domain distillation\" and \"Confidence-Guided Domain Construction\" are at least misleading terms. \n  see line 212: \"The remaining low-confidence samples are randomly and evenly divided into K subsets ...\"\n\n  That does not match any common definition of cross-modality or domain. A modality typically has a structural difference to other modalities, a  domain has at least a difference in probability distribution relative to other domains.  By construction, this is not happening. What they do is, constructing partially overlapping subsets (overlapping due to injection of samples with high similarity) for training autoencoders of the first set on them.\n\nWith this, many parts in the paper need to be rewritten.   \n\n&nbsp;\n\n- They put the central claim in the title of being robust to outliers in the training data. However there is only one  measurement of performance for different contamination noise levels - for VisA and for a very limited range. Given the claim, that would have been a must have experiment for the proposed method and baselines.\n\n&nbsp;\n\n- some smaller but important errors: \n\n    - line 121: \"The training objective is to minimize the cosine distance between teacher and student features across all L = 3 layers on normal samples as\"\n\n   cosine is not a distance. As can be seen in eq(2) it is not minimized but maximized. \n\n    - line 201: \"Specifically, we inject a portion of highly confident normal samples into each domain based on normality confidence scores,\"\n\n    the normal in normal samples suggests that one would use normal data label. That is not the case. It confused here\n\n&nbsp;\n\n- the autoencoder architecture is nowhere described. How was it chosen ? \n\n- as part of this in section 5.3 one cannot understand to what layers {2, 3, 3, 2} belongs to.\n\n&nbsp;\n\n- overall the paper is hard to read. In parts also because things are described in words and defined later. \n\n    - for example line 178 \"we propose Confidence-Guided Domain Construction to extract high-confidence normal\nsamples from the original training set\"\n    - the confidence is defined further down in an equation but not referenced in this place. Same in line 201\n\n&nbsp;\n\n- hyperparameter settings in the appendix evaluated on test data. It is not clear from that how to set them in practice. \n\n&nbsp;\n\n- minor: The most simple additions in Table 4, namely F.P. (= adding noise) and Reg (\"with Reg. introducing regularization by distilling from\nthe previous global student.\") which improves I-AUC , are added at the end. \nThey seem to achieve large gains and this kind of masks the effect of the first set of autoencoders.\n\n- minor: the overlap settings are likely of lesser value. According to the paper they use test time anomalies in training.\n\n- minor: in the most important metric, Image-AUC results are mixed: better on VisA, not better on MVTec-AD\n\n&nbsp; \n\n- no code and weights provide to take a look which given the weaknesses would be helpful.\n\n&nbsp; \n\nOverall, this paper would need one larger iteration and has to be checked after the iteration - this is due to the false methodological and the one misleading claim, which have to be removed (note the removal of reversal distillation (RD) also then requires to remove the discussion section on RD ).\n\nQUESTIONS:\nHow actually Anomaly inference is \"fenced\" off in the submission ?\n\nThe autoencoder architecture is nowhere described. What is its architecture? \n\n\nWhat does that mean:\n\n\"Lastly, Conf.D. refers to\ntraining the global student directly on High-Confidence Domains to learn teacher representations\"\n\n?\n\n\nSince the first set of autoencoders produce features which are selected by the similarity to the second (\"global\") autoencoder, why not using right away the features from  the second (\"global\") autoencoder, but add more noise, and skip the first set ? This would need to be checked in ablation study as well. It might well be that this gives very similar scores.\n\nThis also is supported by Table 4, where F.P. and Reg improves I-AUC a lot,\n\nLIMITATIONS:\nNot much done. However this is not the key issue for this paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to tackle fully unsupervised anomaly detection (with contamination data). This paper pioneer adapting the conventional knowledge distillation (KD) paradigm to the FUAD task. However, due to the presence of anomalies in the training data, traditional KD methods risk learning anomaly representations well during training. To address this, This paper proposes a novel cross-domain distillation framework. First, by domain-specific training on data of lower anomaly ratio can mitigate the student’s tendency to overfit to teacher’s anomaly features. Then, the domain-specific student can produce pseudo-normal features when applied to other domains, cross-domain knowledge aggregation is performed to guide a global student to learn generalized normal representations across all samples.\n\nSTRENGTHS AND WEAKNESSES:\n**Paper Strengths:**\n\n1. The proposed cross-domain distillation framework is novel. I think the idea that domain partitioning and then knowledge aggregation is very interesting and indeed helps to weaken the influence of abnormal samples in training data.\n\n2. The motivation behind the framework is also clear, domain partitioning and cross-domain knowledge distillation align well with these two assumptions.\n\n3. Compared to previous FUAD methods (generate pseudo labels for samples and then train on pseudo-labeled normal samples, which relies heavily on whether the pseudo labels are correct). The method proposed in this article is more impressive, which iteratively weakens the influence of abnormal samples during the training process.\n\n4. The idea of dividing domains and aggregating knowledge is not limited to the knowledge distillation based AD methods, this core idea is general and can be extended to other UAD paradigms.\n\n**Paper Weaknesses:**\n\n**[Major]**\n\n1. Although these two assumptions are not hard to understand, a good method cannot be based solely on the assumptions, and there is a lack of corresponding evidence in the paper to support these two assumptions. If these two assumptions do not hold in real cases, then the proposed framework is not sufficiently reasonable. Therefore, relevant quantitative results or representative examples should be provided for these assumptions to increase their persuasiveness.\n\n2. In line 206, “we use the features output by the global student of the previous epoch as the basis for confidence evaluation”. However, what about the first epoch? For the first epoch, how to obtain the confidence score? This is not clear.\n\n3. The experimental evaluation is not sufficient. This paper only conducts experiments on MVTecAD and VisA datasets with noise ratio 0.1 and 0.05. The authors should conduct more experiments on other datasets (e.g., MVTecAD3D, MVTecAD-LOCO, BTAD, etc) to more comprehensively evaluate the method. And this paper should also conduct experiments with larger noise ratio (e.g., 0.2, 0.3, 0.4) to evaluate the robustness of the method when encountering severe contamination. Additionally, why does VisA use 0.05 noise ratio not the 0.1 ratio used in MVTecAD?\n\n4. In figure 5, the qualitative results are too few, you should add more qualitative results in figure 5.\n\n**[Minor]**\n\nIn the caption of figure 1, “ross-Domain” should be “Cross-Domain”, replacing “MVTec AD-noise-0.1” with “MVTecAD-noise-0.1” is better. In line 237, “. providing” should be “, providing”. In line 286, “MVTec AD-noise-{0.2,0.15}” should be “MVTec AD-noise-{0.02,0.15}” as in figure 3.\n\nI think the idea in this paper is novel and interesting, and will have good inspiration for future FUAD works. It may also become a classic framework in FUAD research. However, due to the above weaknesses, I would like to give a borderline accept. If the author can address these issues well, I am glad to raise the score.\n\nQUESTIONS:\nPlease see Strengths And Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies fully unsupervised anomaly detection where training data include noises. A new algorithm is proposed which first divides the training set into multiple domains with lower anomaly ratios and train a domain-specific student for each. Then, pseudo-normal features are generated by domain-specific students collaboratively guide a global student to learn generalized normal representations across all samples. Experiments on several datasets verify the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed method is technically sound. \n- Experiments on several datasets shows the advantage of the proposed method over existing methods. \n- The paper is well presented. \n\nWeakness \n- There misses an important baseline. If the key motivation to study the fully unsupervised anomaly detection is to fully utilize the training data, an important baseline would be a method trained with noise-free data. Only by comparing a baseline method like this with demonstrated improvement, the benefit of training with noisy data can be justified. However, I did not see such a baseline employed for comparison.\n\n- The two key assumptions lying the foundation of the proposed method have not been enough validated through rigorous proofs or specific experiments. While indeed there are some analytic reasoning for the first assumption, there is only some explanatory descriptions for the second assumption. From the perspective of the experiments, the final results indeed partially validate the soundness of the two assumptions, it is still necessary to design some experiments to validate each assumption separately. \n\n- Which is the effect of further increasing the noise level? Will the proposed method show greater advantages over existing methods.\n\n-  The proposed method employed a power teacher model, which gives the proposed method additional advantage over the existing methods. \n\n- The improvements over the existing method are not significant. \n\n- The performance difference of different design in the ablation study is very small, hard to validating the effectiveness.\n\nQUESTIONS:\nSee the weakness above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a practical and challenging task: fully unsupervised anomaly detection (FUAD), where the training set contains unlabeled anomaly samples. The authors point out that under this setting, traditional knowledge distillation (KD) methods run the risk of over-simulating anomalous features in the training set by the student network, leading to degradation of the detection performance. To solve this problem, the paper proposes a framework called Cross-Domain Distillation (CDD). The framework consists of two core stages: 1) Domain-Specific Training (DST), which divides the dataset into multiple domains with low anomaly ratios and trains a specialized student network for each domain to reduce the learning of anomaly patterns; 2) Cross-Domain Knowledge Aggregation (CDKA), which is a method to reduce the learning of anomaly patterns; and 3) Cross-Domain Knowledge Aggregation (CDKA), which is a method to reduce the learning of anomaly patterns. Aggregation (CDKA), which utilizes student networks not trained on a specific domain to generate “pseudo-normal” features for samples in that domain, and uses this to guide a domain-wide network of students to learn a generalized normal pattern representation. The experimental results show that this method achieve SOTA performance on the problem of FUAD, and ablation study demonstrates the effectiveness of proposed framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n\n1. The writing is clear and easy to follow.\n2. The proposed cross-domain distillation idea is intuitive.\n3. The experimental design is rigorous.\n\nWeaknesses.\n\n1. The improvement of performance is limited: The performance gains of the proposed CDD framework, particularly in key metrics like Image-level AUC on the MVTec AD dataset, are marginal when compared to existing state-of-the-art FUAD methods (e.g., SoftPatch). This limited improvement does not seem to fully justify the significant increase in model complexity, which involves multi-stage training with multiple student networks and dynamic parameter scheduling. The modest gains challenge the practical value of adopting such a complex architecture over simpler, yet competitive, alternatives.\n\n2. The final loss function for the global student is ambiguous: The paper introduces two separate loss terms for training the global student: Eq. 20 for cross-domain pseudo-normal feature distillation and Eq. 21 for distillation on high-confidence samples. However, the manuscript fails to specify how these two loss components are combined into a single, final objective function for optimization. It is unclear whether they are simply summed or combined with weighting coefficients. This omission is a critical flaw as it hinders the reproducibility of the work, which is a cornerstone of scientific research.\n\n3. Justification for the feature layer fusion strategy is lacking: While the paper specifies the method for fusing layer-wise anomaly maps in Eq. 11, it provides no justification for this design choice. Different network layers capture features at varying semantic levels, and it is not self-evident that a simple, unweighted sum is the optimal fusion strategy. The authors do not discuss or ablate alternative approaches, such as weighted summation or taking the maximum value, which leaves a gap in the methodological rigor and the exploration of a key design component.\n\nQUESTIONS:\n1. Please provide the exact formulation of the final loss function for the global student and justify this specific design choice.\n2. The feature fusion strategy in Eq. 11 employs a simple summation across layers. Please provide an ablation study or a stronger justification for why this unweighted summation is preferred over other potential fusion techniques, as different layers capture distinct feature semantics.\n3. On page 6, the paper claims that the confidence-guided strategy ensures \"The normal distribution in each domain remains more similar to the overall normal distribution.\" However, this crucial claim is not substantiated with either theoretical analysis or empirical evidence. Please provide a justification for this statement.\n4. The framework's effectiveness relies on Assumption 2. However, this assumption might be fragile. In scenarios where different anomaly types across domains share subtle, underlying features (e.g., similar textural defects or color shifts), could the \"pseudo-normal\" features generated from other domains become contaminated, thereby misleading the global student? Please discuss the robustness of your method to such violations of this core assumption.\n5. The Consensus-driven Selection strategy (Eq. 17 & 18) selects the pseudo-normal supervision signal most similar to the features from the previous epoch's global student. This creates a potential risk of confirmation bias, where the global student is primarily guided towards patterns it already models well. This could prevent it from learning novel aspects of the true normal distribution or correcting its past errors, potentially leading to convergence in a sub-optimal local minimum. How does your framework mitigate this risk of self-reinforcement?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper is the first to explore the application of the knowledge distillation paradigm in fully unsupervised anomaly detection tasks. It addresses the over-generalization problem present in traditional knowledge distillation methods through two key strategies: Domain-Specific Training and Cross-Domain Knowledge Aggregation. Experimental results demonstrate its effectiveness.

STRENGTHS AND WEAKNESSES:
**Strengths**

1.	This is the first work to explore the application of the knowledge distillation paradigm in fully unsupervised anomaly detection tasks.
2.	The presentation of the paper is relatively clear.
3.	The paper provides corresponding explanations for its assumptions.

**Weaknesses**

1.	The performance improvement compared to SoftPatch in Table 1 seems to be limited.
2.	The paper lacks evidence to support some of the viewpoints presented. For instance, the issue mentioned in lines 48-53, i.e., “This issue becomes more pronounced under the FUAD setting”, if supported by a simple experiment, I believe such evidence could further reinforce the motivation of the paper.

QUESTIONS:
1.	In lines 34-37, it is mentioned that the 'Knowledge Distillation paradigm offers a storage-efficient alternative.' Could the authors provide relevant data on the storage overhead?
2.	In lines 56-57, the authors present their intuition. Could the authors justify the validity of this intuition from a theoretical or experimental perspective?
3.	Could evidence be provided to support the validity of the hypothesis presented in lines 62-63?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03