Title: ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: Anomaly Detection

Anomaly Representation Pretraining
Abstract: The current mainstream and state-of-the-art anomaly detection (AD) methods are
substantially established on pretrained feature networks yielded by ImageNet pre-
training. However, regardless of supervised or self-supervised pretraining, the
pretraining process on ImageNet does not match the goal of anomaly detection
(i.e., pretraining in natural images doesn’t aim to distinguish between normal and
abnormal). Moreover, natural images and industrial image data in AD scenarios
typically have the distribution shift. The two issues can cause ImageNet-pretrained
features to be suboptimal for AD tasks. To further promote the development of
the AD field, pretrained representations specially for AD tasks are eager and very
valuable. To this end, we propose a novel AD representation learning framework
specially designed for learning robust and discriminative pretrained representa-
tions for industrial anomaly detection. Specifically, closely surrounding the goal
of anomaly detection (i.e., focus on discrepancies between normals and anoma-
lies), we propose angle- and norm-oriented contrastive losses to maximize the
angle size and norm difference between normal and abnormal features simulta-
neously. To avoid the distribution shift from natural images to AD images, our
pretraining is performed on a large-scale AD dataset, RealIAD. To further alle-
viate the potential shift between pretraining data and downstream AD datasets,
we learn the pretrained AD representations based on the class-generalizable repre-
sentation, residual features. For evaluation, based on five embedding-based AD
methods, we simply replace their original features with our pretrained represen-
tations. Extensive experiments on five AD datasets and five backbones consis-
tently show the superiority of our pretrained features. The code is available at
https://github.com/xcyao00/ADPretrain.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a feature extractor grounded in existing pretrained model (such as ResNet50, CLIP on ImageNet) for industrial anomaly detection (IAD). The proposed framework designs an angle- and norm-oriented contrastive loss to train the Feature Projector based on RealIAD, a large industrial anomaly detection dataset. The empirical results indicate that the features produced by the trained Feature Projector improve the detection performance of five embedding-based AD methods in comparison to using their original features.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The proposed framework effectively utilizes the RealIAD and the existing pretrained feature extractors for industrial anomaly detection.\n\n2. The empirical results demonstrate the effectiveness of the proposed framework.  \n\n\n**Weaknesses:**\n\n1. Ambiguous and misleading writing and statements\n\n(1)\tThe “Unsupervised Anomaly Detection” is highlighted in Title of the manuscript. However, the proposed framework is a feature extractor, does not direct relationship with “Unsupervised learning” and the training process of the Feature Projector is based on ground truth of RealIAD (supervised learning). In fact, the proposed method is only designed to obtain the features and it doesn’t care about the specifics of the downstream AD task, whatever supervised or unsupervised.\n\n(2)\tThis work only focuses on industrial anomaly detection (IAD). Thus, the research field should be clarified rather than directly using “anomaly detection” in the beginning. For instance, the description in line 1-3 is incorrect when considering other data types (time-series, graph, tabular) or application scenarios.\n\n(3)\tThis work highlights the “Pretraining” in the many places of the manuscript, which makes me think a totally novel pretrained model for IAD in the beginning. However, the experimental settings show that the backbone (architecture) and even the parameters of the pretrained models are remained accurately. In other words, this work just fine-tunes the existing pretrained models on a vertical field dataset (RealIAD) for industrial anomaly detection via slotting a Feature Projector. I suggest that authors revise some statements.\n\n2. Unclear technique details\n\n(1)\tOn Angle-Oriented Contrastive Loss: \nThe training process of Feature Projector is supervised, that is it is easy to get positive-pairs and negative-pairs, why still using data augmentation to produce more positive-pairs? And, how to ensure the augmented samples closer to normal data than abnormal data? Especially, if the forming of anomalies is similar with the used data augmentation techniques, for example, food spoilage and random color jitters, it may cause the wrong optimization direction.\n\n(2)\tNorm-Oriented Contrastive Loss:\nWhy using pseudo-Huber distance? Any defects or problems if using a simple Euclidean? I understand the first term of the loss, but why adding a multiplicative term $e^{d_j}$?\n\n(3)\tLearnable Key/Value Attention and Feature Projector:\nThe current contents are hard to understand in Section 3.3. To improve the readability, it should describe the function of feature projector using standard formulars step by step.\nI am curious about the learnable reference. After training, whether it was be fixed for extracting features for new datasets? How to understand the learnable reference?\n\n(4)\tAuthors need to check and clarify the optimized object. In line140, which one is the residual representation, x or r? And all the loss functions use x as optimized object. \n\n3. The fairness of comparison\n\nIn Appendix D Hyperparameter Sensitivity, authors explore the effects of different hyperparameter settings for detection performance. There are several concerns as follows:\n\n(1)\tWhat and where is the test data?\n\n(2)\tIt is unreasonable that the hyperparameters selection depends on the performance on the used embedding-based AD methods. The proposed framework is a feature extractor and it should be method-agnostic, like a pretrained ResNet-50 on ImageNet rather than method-dependent, otherwise, the proposed framework is just tailored for certain AD methods.\n\n(3)\tWhy does the performance look like quite robust (<1% fluctuation) in almost all cases for the changing of the hyperparameters?\n\n(4)\tFor selection of $N_r$, I noticed the performances are quite close for $N_r=256$ and $N_r=2048$. It should extend the range from 1-256 to more suitable selection.\n\n4. Statistical significance and reproducibility of the results\n\n(1)\tThe empirical results do not report any average values and standard variance. \n\n(2)\tThe code is not provided.\n\nQUESTIONS:\n1. How to determine the hyperparameter radius r? Authors have mentioned that it was set to 0.4 in the experiments.\n\n2. How large is the feature dimension? Does different dimension have a significant impact on detection performance?\n\n3. It needs to be clarified how to construct a few-shot AD method by the feature norm. \n\nOther questions see Weaknesses\n\nLIMITATIONS:\n1. Building on the optimization objective and empirical results, the features produced by the proposed framework show the strong distinguishability. In such situations, it should be explored when directly applying the feature vectors to the classical AD algorithms, such as OCSVM, LOF, Isolation Forest, KNN etc. It is possible to obtain good performance due to the results of FeatureNorm. If the features indeed achieve better or comparable performance on the conventional AD algorithms than other features even the end-to-end deep methods, the proposed framework will produce much higher social impact due to the robustness and low computational cost of these conventional methods.\n\n2. Two simple Feature Projector designs should be explored, (1) concatenating a MLPs (learnable parameters) behind the pretrained model (fixed parameters) and then fine-tuning the MLPs; (2) fine-tuning only the readout layer of the pretrained model when freezing all other parameters", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new contrastive learning strategy and feature adaptation network focusing on industrial anomaly detection scenarios. The ability of residual features is fully exploited, and Learnable Key/Value Attention is proposed for feature adaptation.\nThen, Angle-Oriented Contrastive Loss and Norm-Oriented Contrastive Loss are designed to expand the distinction between normal and abnormal features. The sufficient experimental performance improvement on five public datasets proves the advantages of this method. And it can also enhance the adaptability of different types of feature extractors to industrial scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors propose two novel contrastive learning losses, Angle-Oriented Contrastive Loss and Norm-Oriented Contrastive Loss, and the theoretical analysis of the two losses is very thorough.  The ablation studies and t-SNE prove that these two losses could increase the difference between normal and abnormal features, thereby improving the performance of the current anomaly detection model.\n2. Although residual features are not proposed for the first time in this paper, the analysis and application of residual features in this paper are thorough.  Moreover, the paper designs an effective Learnable Key/Value Attention to transform this feature for subsequent representation learning.\n3. The authors conduct sufficient experiments, proving that this training strategy could bring improvements for different AD methods and feature extractors of different architectures.\n4. The issue focused on by this paper has great significance to the industrial field. This work could fill the gap in open-source work on feature adaptation in the industrial AD tasks.\n\nWeaknesses:\n1. The ablation studies in Table 2 indicate that the non-fixed feature extractor leads to significant performance degradation. This phenomenon means that the weights obtained through pre-training in natural scenarios are very important. This work only adapts the original pre-trained features to industrial scenarios, which is difficult to be defined as \"pretrain\".\n2. line73: \"this is the first study dedicated to anomaly representation pertaining.\" However, the paper **spot-the-difference**[1] proposes a special data augmentation strategy on natural images and uses contrastive learning for pre-training for subsequent industrial anomaly detection. Although the method proposed in this paper is quite different from spot-the-difference, it is beneficial to discuss it in the related works.\n3. Why is only the GLASS-h variant used for evaluation? Is it because the Manifold Hypothesis in GLASS-m contradicts the Hypersphere-based loss in this paper?\n4. The products in the Real-IAD dataset are all \"object classes\". Therefore, providing some qualitative and quantitative results on the \"texture classes\" (such as carpet, tile and grid on the MVTec AD dataset) will be beneficial for better evaluating the generalization performance of features.\n5. line 188: \"we can downsample the mask to get feature labels.\" What is the specific downsampling operation (max pooling or average pooling or others) ? Different downsampling strategies could lead to some patches being assigned different labels.\n6. Are the output features of the Feature Projector used to calculate both the Angle-Oriented Contrastive Loss and the Regul-oriented Contrastive Loss?  The symbols in this paper are not consistent and clear.\n7. In the Learnable Key/Value Attention, the \"addition operation\" are replaced to \"subtraction operation\". This improvement requires more explanations and motivations. It seems like performing a secondary residual calculation on the original residual features, but the original residual features have already highlighted the abnormal regions.\n8. In the inference phrase, some normal samples need to be provided for calculating residual features, which might limit the wider application of this framework, such as zero-shot and contaminated AD tasks without labeled normal data.\n9. In few-shot (especially 2-shot) settings, the influence of reference samples on residual features needs to be discussed. For example, for the screw class of the MVTec AD dataset, samples with different rotation directions are adopted as reference samples. Or, reference samples could be selected through different random seeds for sensitivity analysis.\n\nSome extended problems:\n1. Whether the features extracted through the framework in this paper will eliminate the original pre-trained model's ability to distinguish different types of anomalies.\n2. Whether it can be applied to supervised anomaly detection tasks by adding a classification head or a segmentation head, such as the experiments in the paper \"Spot-the-difference\". Although abnormal images are difficult to collect, in real industrial scenarios, supervised anomaly detection models still have very wide applications at present.\nThe responses to these extended questions will not lead to my negative evaluation of this paper. This paper has provided enough contribution to the AD community.\n\n[1] Zou Y, Jeong J, Pemula L, et al.  Spot-the-difference self-supervised pre-training for anomaly detection and segmentation[C]//European Conference on Computer Vision.  Cham: Springer Nature Switzerland, 2022: 392-408.\n\nQUESTIONS:\nSee weaknesses.\n\nIf the authors could solve my questions, especially weaknesses 2, 4, 7 and 9, I will consider further improving my rating.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the intriguing and under-explored topic of anomaly representation pretraining, a critical yet scarcely researched area in unsupervised anomaly detection. Existing state-of-the-art AD methods heavily rely on ImageNet-pretrained features, which suffer from mismatched objectives (e.g., classification vs. anomaly discrimination) and distribution shifts between natural and industrial images. The authors propose ADPretrain, a novel framework that leverages the large-scale RealIAD dataset to learn robust pretrained representations by integrating residual features and designing angle- and norm-oriented contrastive losses. This approach aims to maximize discriminative differences between normal and abnormal features while mitigating distribution shifts.  While the study acknowledges limitations such as reliance on embedding-based AD methods and fixed backbone networks, its focus on task-specific pretraining fills a significant research gap. By pioneering anomaly-focused pretraining, the work offers valuable insights and paves the way for future advancements, making it a promising contribution despite its room for improvement.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**  \n- The paper addresses the limitations of using ImageNet-pretrained features for anomaly detection by proposing a task-specific pretraining framework, with well-designed residual features and contrastive losses tailored for AD.\n\n- The related work section provides a thorough review of existing methods and clearly positions the novelty of the proposed approach.\n\n\n### **Weaknesses**  \n-  While the paper emphasizes that ADPretrain outperforms ImageNet-pretrained features, it does not explicitly compare its framework with state-of-the-art fine-tuning methods like FYD or mean-shifted contrastive loss. Such a comparison is crucial to validate whether pretraining truly offers advantages over adaptive fine-tuning strategies in addressing distribution shifts .  \n\n-  The experiments do not test the framework under more settings, such as noisy conditions where normal samples might be corrupted. For example, methods like SoftPatch [1] have shown robustness to noise, but ADPretrain’s reliance on residual features—derived from clean normal references—may struggle if training data contains noise. This gap leaves uncertainty about its performance in real-world scenarios with imperfect data.  \n\n-  It does not evaluate modern architectures like EfficientNet or DINOv2, which have demonstrated superior representation capabilities in self-supervised learning. Additionally, while the paper mentions that residual features are class-generalizable, it does not explicitly validate their effectiveness on cutting-edge backbones (e.g., UniFormaly’s findings on ViT vs. CNN [2]). Expanding to more architectures would strengthen the generalizability claim.  \n\n[1] Jiang X, Liu J, Wang J, et al. Softpatch: Unsupervised anomaly detection with noisy data[J]. Advances in Neural Information Processing Systems, 2022, 35: 15433-15445.\n\n[2] Lee Y, Lim H, Jang S, et al. Uniformaly: Towards task-agnostic unified framework for visual anomaly detection[J]. Pattern Recognition, 2025: 111820.\n\nQUESTIONS:\n- Compare ADPretrain with fine-tuning methods (e.g., FYD, PANDA) on the same datasets to quantify the benefit of task-specific pretraining.  \n- Introduce synthetic noise or corrupted samples in experiments to assess robustness.  \n- Validate the framework on emerging backbones (EfficientNet, DINOv2) to confirm its applicability across modern architectures.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ADPretrain, an anomaly-aware pretraining framework designed to generate better feature representations for unsupervised anomaly detection. The authors propose a pseudo-anomaly discrimination task along with multi-scale feature fusion to make the pretraining more aligned with downstream AD tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe key strengths of this paper is that it tackles a limitation in the current AD landscape.\nThe results are good. Across a wide variety of datasets and multiple backbones, the method improves performance. \n\nWeaknesses\n\n1.\tThe paper lacks direct comparisons against alternative pretrained backbones beyond ImageNet (e.g., MAE, DINOv2, self-distillation), which makes it difficult to contextualize the performance gain solely to the proposed pretraining strategy.\n2.\tThe method essentially acts as a drop-in feature encoder, and while the performance improvements are notable, the core contribution is mostly centered around representation learning rather than architectural novelty.\n3.\tThe paper would benefit from further analysis of computational cost, especially regarding whether ADPretrain can reduce downstream training/inference cost or improve sample efficiency.\n\nQUESTIONS:\nCould the authors compare ADPretrain against other pretrained representations such as MAE, DINO, or industrial-specific SSL models?\n\nWhat is the impact of pretraining dataset choice on generalization? Is ADPretrain truly domain-agnostic?\n\nGiven the add-on nature of ADPretrain, is it possible to integrate it into fully end-to-end AD pipelines (e.g., UniAD), rather than only use it for feature extraction?\n\nCould the authors elaborate on whether their method helps in resource-constrained scenarios (e.g., faster inference, fewer training images)?\n\nLIMITATIONS:\nThe authors have clearly acknowledged the main limitations of their work, including the fixed backbone design and the restricted applicability to embedding-based AD methods. These reflections are fair and sufficient given the scope of the paper", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}