ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining
Authors: Xincheng Yao
Yan Luo
Zefeng Qian
Chongyang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Anomaly Detection
Anomaly Representation Pretraining
Primary area: applications
TL;DR: We propose a novel AD pretraining framework specially designed for learning robust and discriminative pretrained representations for industrial anomaly detection.

Abstract: The current mainstream and state-of-the-art anomaly detection (AD) methods are
substantially established on pretrained feature networks yielded by ImageNet pre-
training. However, regardless of supervised or self-supervised pretraining, the
pretraining process on ImageNet does not match the goal of anomaly detection
(i.e., pretraining in natural images doesn’t aim to distinguish between normal and
abnormal). Moreover, natural images and industrial image data in AD scenarios
typically have the distribution shift. The two issues can cause ImageNet-pretrained
features to be suboptimal for AD tasks. To further promote the development of
the AD field, pretrained representations specially for AD tasks are eager and very
valuable. To this end, we propose a novel AD representation learning framework
specially designed for learning robust and discriminative pretrained representa-
tions for industrial anomaly detection. Specifically, closely surrounding the goal
of anomaly detection (i.e., focus on discrepancies between normals and anoma-
lies), we propose angle- and norm-oriented contrastive losses to maximize the
angle size and norm difference between normal and abnormal features simulta-
neously. To avoid the distribution shift from natural images to AD images, our
pretraining is performed on a large-scale AD dataset, RealIAD. To further alle-
viate the potential shift between pretraining data and downstream AD datasets,
we learn the pretrained AD representations based on the class-generalizable repre-
sentation, residual features. For evaluation, based on five embedding-based AD
methods, we simply replace their original features with our pretrained represen-
tations. Extensive experiments on five AD datasets and five backbones consis-
tently show the superiority of our pretrained features. The code is available at
https://github.com/xcyao00/ADPretrain.

ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining
Xincheng Yao', Yan Luo Zefeng Qian', Chongyang Zhang 1.2+
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University
MoE Key Lab of Artificial Intelligence, AT Institute, Shanghai Jiao Tong University *College of Artificial Intelligence, Nanjing Agricultural University Key Laboratory of Livestock Farming Equipment, Ministry of Agriculture and Rural Affairs, Nanjing Agricultural University {i-Dover, zefeng-qian, sunny SaanmyZZhng)0sjte edu cn luoyan@njan edu. cn
Abstract
The current mainstream and mate-of-the--ee anomaly detection (AD) methods are substantially established on pretrained feature networks yielded by ImageNet pre- training. However, regardless of supervised or self-supervised pretraining, the pretraining process on ImageNet does not match the goal of anomaly detection (i.e., pretraining in natural images doesn't aim distinguish between normal and abnormal). Moreover, natural images and industrial image data in AD scenarios typically have the distribution shift. The two issues can cause ImageNet-pretraine features to be suboptimal for AD tasks. To further promote the development of the AD field, pretrained representations specially for AD tasks are eager and very valuable. To this end, we propose a novel AD representation learning framework specially designed for learning robust and discriminative pretrained representa- tions for industrial anomaly detection. Specifically, closely surrounding the goal anomaly detection (i.e., focus on discrepancies between normals and anoma- lies), we propose angle- and norm-oriented contrastive losses to maximize the angle size and norm difference between normal abnormal features simulta- neously. To avoid the distribution shift from natural images to AD images, pretraining is performed on a large-scale AD dataset, RealIAD, To further alle- viate the potential shift between pretraining data and downstream AD datasets, we learn the pretrained AD representations based on the class glass-generalizablle repre- sentation, residual features. For evaluation, based on five embedding braedddig--aeedd based AD methods, we simply replace their original features with our pretrained represen- tations. Extensive experiments tently show the superiority of our pretrained features. The code available at https://gitthb. wttp:///ww/occcccccccooaa SateyyaacoO/APPetraaai.
five AD datasets and five backbones consis-
Introduction
In recent years, the anomaly detection (AD) field has undergone rapid evolution.
Researchers
have proposed various deep AD methods, including reconstruction-basee 51
[50
48 16]. feature-
distance-based 30 33 13
(OCC)
[36]
24
49 23 distillation-bant
37 9 53), and snterativeemodll-baaeee AD methods [12
20
14
52
10 etc. Although different
taxonomies of methods have their distinctive insights and collaboratively address the challenging issues in anomaly detection from various perspectives, the essence of well-performed AD methods
Both Yan Luo and Chongyang Zhang are Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Separcaaioo
Abanding Mehoes
AETUP Dinte
Frb
Priteied
0,466
Nove There
$0.00
Nel
0129
Figure 1: (a) Conceptual illustration of anomaly representation pretraining. (b) Performance compari- son on MVTecAD (left) and VisA (right). "w/o" and "w/" refer to without and with our pretrained features. Under multiple AD methods and backbones, our pretrained features are consistently superior to the original features (dashed lines are overall on top of solid lines).
can still be attributed to the representation ability of features, i.e., if normal and abnormal features are highly discriminative (e.g., linearly separable), anomalies can be easily detected out as outliers.
Due to the scarcity and uncertainty of anomalies, the main paradigm in AD is unsupervised (with only normal samples for training). This paradigm makes it hard to learn highly discriminative representations for two reasons: (1) With only normal samples, classical AD methods resort to self-supervised proxy tasks (e.g., auto-encoding reconstruction) learn normal representations, assuming that models would fail in abnormal image regions. However, only learning normal samples may easily lead the phenomenon of "pattern collapse" [50 36 where both normal and abnormal features are similar. E.g., anomalies may be well reconstructed in reconstruction-baat based methods. (2) The scale of conventional AD datasets is usually not too large, relying on limited normal training data for representation learning would also restrict the quality of learned representations.
To overoome the weaknesses of seff-supervised representation learning from scratch, previous works 33 (starting with ( have confirmed that utilizing pretrained features yielded by ImageNet pretraining can significantly improve AD performance compared to features learned from scratch on the AD datasets. Afterward, mainstream and state-of-the- AD methods are almost based on ImageNet-pretraines feature networks. However, maageNet-pretraineg
features are still suboptimal for AD tasks for two reasons: (1) The conventional pretraining ways (e.g., image classification, image contrastive learning, masked image modeling) don't meet the goal of anomaly detection, as there is no concept of normal and abnormal during the ImageNet pretraining process. (2) Natural images in ImageNet and image data in AD scenarios typically have the remarkable distribution shift, without further adaptation, the performance of pretrained features on AD data may be constrained [39.. Therefore, to further advance the development of anomaly detection, it's necessary to construct and learn pretrained representations specially for anomaly detection tasks.
In this paper, we explore the problem of anomaly representation pretraining, where the objective is to learn pretrained AD representations that are better than Imageeett-retrrined features when applied to downstream AD methods (see Fig Tp. Like conventional pretraining works, we expect that anomaly representation pretraining would be performed on large-scale AD dataset. Luckily, the proposal of the RealIAD dataset (44) provides us with such a prerequisite. RealIAD has a large enough data scale, containing total of 151050 images, of which 99721 images are normal and 51329 images are abnormal. Based on RealIAD, we propose a novel AD representation learning framework specially designed for learning robust and discriminative pretrained representations for industrial anomaly detection. By pretraining on RealIAD, we can avoid the distribution shift from natural images to industrial AD images. As the main characteristic of AD tasks is to focus on the discrepancies between normals and anomalies, contrastive learning should be the most suitable pretraining paradigm. For better representation learning, we propose angle- and norm-oriented contrastive losses to maximize the angle size and norm difference between normal and abnormal features simultancously. As from the feature similarity perspective, differences between two features are reflected in angle and norm. Further considering the potential distribution shift caused by different product categorics between RealLAD and downstream AD datasets, we leam the pretrained AD representations based on the
'The preetraindd is to refer to feature networks pretrained on all sorts of large-scale datasets (not just ImageNet). As ImageNet is the most well-known pretraining dataset, we still use "ImageNet- pretrained". "reerre....eetriinee.....

---

generalizable representation in anomaly detection, residual features [46]. We find that when used as pretrained AD representations, residual features are better than vanilla features extracted by the feature network. Our contributions are as follows:
1. To the best of our knowledge, this is the first study dedicated to anomaly representation pretraining. We construct an AD representation learning framework, which can learn robust and discriminative pretrained features specially for anomaly detection tasks.
2. To fully optimize the discrepancies between normal and abnormal features, we propose angle- and norm-oriented contrastive losses, which can maximize the angle size and norm difference between two features simultaneously.
3. In five embedding- AD methods, we replace their original features with our pretrained features. Extensive experiments show that our pretrained features can consistently surpass the original features. Moreover, another merit of our pretrained features is that they are also good representations for few-shot anomaly detection by simply using feature norms as anomaly scores.
2 Related Work
Anomaly Detection With Pretrained Features. The current mainstream and state-of-the-art AD methods mostly based on Saaggeeet-pretrainee feature networks. Starting with (I), the authors found that simply combining pretrained features with the KNN algorithm can significantly outperform previous self supervised methods. Afterward, Image:Net-pretaanne features widely adopted by AD methods (we call them embedding- based AD methods). Representative methods include: PaDiM and PatchCore. PaDiM S extracts pretrained features to model Multivariate Gaussian distribution and then utilizes the Mahalanobis distance to measure anomaly scores. PatchCore [3] proposes to utilize locally aggregated features and introduces maximally representative memory bank of normal features. However, images in AD scenarios generally have an obvious distribution shift with natural images. To better account for the distribution shift, subsequent methods mainly follow a standard paradigm: fixing pretrained feature networks and designing learnable modules. Representative
methods include: embedding- based reconstruction methods 45 50 16). distillation-bar -based methods
3 37
533, and normalizing- weallliziinglll----e flow -bsed methods [3
12
35
47
Feature Adaptation to AD Tasks. Due to the distribution shift, pretrained features still suboptimal for AD tasks. To better utilize pretrained features, this line of work aims to adapt pretrained features to the target distribution in AD datasets. However, this doesn't mean simple fine-tuning is feasible, as naive fine-tuning in AD datasets often results in catastrophic collapse (feature deterioration) and reduced performance (27 30). To this end, PANDA (30 proposes techniques based on early stopping and EWC (22). a continual learning method, to mitigate the catastrophic collapse. In (31] Reiss, et al. propose a contrastive method, mean-shifted contrastive loss, which is more suitable for AD tasks by ensuring the compactness of normal features after fine-tuning, In FYD [54]. the authors propose a dense non-contrastive learning method for self-supervised learning of compact normal features and utilize a soop-gradient strategy to alleviate the collapse. In (32), Rippel, et al. propose a Gaussian fine-tuning method, where the pretrained network is fine-tuned by minimizing the Mahalanobis distance to the estimated Gaussian distribution. However, these methods are still not general, as their fine-tuned feature networks are applicable only to the trained dataset. By comparison, our work takes a step forward beyond fine-tuning, which is dedicated to learning robust and discriminative pretrained features for downstream AD tasks. More discussions are in Appendix B.
3 Method
Overview. Our proposed anomaly representation learning framework is illustrated in Fig 2 Our pretrained AD representations are based on residual features. During pretraining, the residual features are transformed into a latent space by the Feature Projector (Sec(3.3), and then optimized by the angle- and com-oorietedd contrastive losses (.2) to maximize the angle size and norm difference between normal and abnormal features simultaneously. As our method aims to learn better representations for AD tasks, its usage is the same conventional feature extractors. We can simply
'In this paper. "embedding- based" is broad concept. We use embedding based AD methods to refer to all sorts of AD methods that utilize pretrained feature networks.
3

---

incorporate our pretrained features outputted by the Feature Projector into existing smbedding-basee AD methods replace the original features.
Backbone Network
Feature Projector
I0-0 de
Outpult
Abnornal
Narmal
+
Frown
Gause Aurr
F Feed Forward MLP
Lcamobic
Contrastve Training
Resizual
Resicual Ope
Rus dual
Subnect
OCS
Oss
Mulli-eed Cross Attention
UH d
|I-1 BlokkL
Bl
Layhr
Backbone Network
Input Festures
Lesmmble Reference
Normal Aemercce Siarples
Figure 2: Framework overview. We learn pretrained AD representations based residual features, while not the features directly produced by the backbone network. Residual features are generated by subtracting normal reference features (Sec which extracted from the normal reference samples. Features yielded by the Feature Projector are optimized by the angle- and norm-one corm-oriented contrastive losses (Sec 3.2). The Feature Projector is based on the Transformer architecture, but alter self eeff-attention to our proposed learnable key/value attention (Sec 3.3).
3.1 Construction of Fundamental Anomaly Detection Representations
first consider what kind of representation features are suitable as pretrained AD representations. We expect that pretrained features can serve as fundamental (general) features in anomaly detection (i.e., can perform well on various AD datasets). Therefore, we think that it's best for pretrained representations be comain-invariant. namely, datasets differ from that in the pretraining AD dataset, the normal and abnormal representation distributions are consistent (relatively invariant) with those during pretraining. To this end, utilize the recently proposed stass-generalizable representations in anomaly detection, residual features [46), the fundamental AD representations and learn pretrained features. As the authors explained and verified in [46] that residual features can be regarded as class-invarian representations compared to the features directly produced by the feature extractor (please refer to [46] for more explanations), we also feel that residual features are potential to be general representations in anomaly detection.
if the data distributions in downstream AD
Specifically, residual features are obtained by matching and then subtracting. For an input image xx x3, backbone network is utilized to extract multi-level (i.e., L) features. For each feature 2// E Bt level I and position (h, w). will match it with the nearest normal reference feature x,, = argmin ||a//..0 - æ1|2 from the corresponding reference feature bank. The R. = (= hh/h/ll///l/t (I,...,
L/K]] is the feature bank for I-th level, where denotes the i-th normal sample and K is the number of normal reference samples. The residual representation of <th.ar is defined as: rh... = 2///0 - During pretraining, for each training sample, we randomly select reference samples from the normal set to increase the diversity of residual features.
Discussions with ResAD. Compared to ResAD, we mainly take advantage of the residual features proposed in ResAD. However, our work and ResAD are clearly two different works, with differences in motivations, concerned problems, and implementation methods. Due to page limitation, we provide detailed difference discussions between our work and ResAD Appendix B.4
3.2 Contrastive Losses for Anomaly Representation Pretraining
We know that one main characteristic of tasks focus on discrepancies between normals and anomalies. Thus, good representation features for anomaly detection should be able to ensure suffi- cient discrepancies between normal and abnormal features. Naturally, learn such representations, contrastive learning should be the most suitable pretraining paradigm (constructing normal- abbooll

---

contrastive pairs). From the feature similarity perspective, the discrepancies between two features are embodied in two aspects: the angle size between two features and the feature norm difference (iee, mathematically, a feature is a vector, differences between two vectors are reflected in angle and norm). Therefore, we propose angle- and norm-oriented contrastive losses to maximize the angle size and norm difference between normal and abnormal features simultaneously. In the following losses, we use x to represent residual features instead of as I is most commonly used to represent features.
Angle-Oriented Contrastive Loss. This loss follows the classical contrastive loss, called InfoNCE 42 loss. Following common practice [5 7 in each training step, we randomly sample a mini-bater of B images and randomly augment each image to obtain an augmented image, resulting in 2B data samples (i.e., for i-th sample, the index of its augmented sample denoted as i' i + B). In this work. we sequentially apply three simple augmentations: random color jitters, random gray scale, and random Gaussian blur. Then, we employ the backbone network to extract features for the 2B samples, with a total of 2NF features. The i-th (i E 21, .)) feature 2; and the feature from the same position in the augmented image, denoted as Fy, 1' = i + N, will be treated as a positive pair. The other 2(N - 1) features within the mini-batch are treated as negative examples. Then, the InfoNCE loss for a positive pair (æi, (x)) is defined as:
exp(sim( 22)//
Cace Cnoe(et, xV) = log
(I)
2N R= [(s
(2)
where [ E 10 is an indicator function evaluating to 1 iff k F i, T denotes a temperature parameter, and sim(2,, xyy) = x/Tx>/|<z; /||=; ||| ||2|| ||2 is the cosine similarity between I; and zy- However, the conventional contrastive learning objective is not suitable for anomaly representation pretraining, as negative examples contain both normal and abnormal features. When z; belongs to normal features, only abnormal features should be treated as negative examples to enlarge the discrepancies between normal and abnormal features. In addition, in mathematics, features can be viewed as vectors starting from the origin. Cosine similarity in fact measures the angle between two features with the origin the center. However, previous work [31] indicated that the standard contrastive loss prefers to optimize features uniformly distributed on the origin-centered sphere, but it is not conducive to making normal and abnormal features more discriminative. This is harmful for anomaly detection. Inspired by (31], we measure the angle between two features with respect to the center of the normal features rather than the origin. Let us denote the center of all normal features in the training set by c = Erexx...maa [z:]. Then, for each feature 2;, we first subtract the center to form center-shifted feature ã. = I2 e with the center c as origin and then calculate the cosine similarity sim( F.). In this way, the contrastive loss preserves features' distances to the center e while maximizing the angles between normal and abnormal features. For positive pair (a,, F.)), the angle-oriented contrastive loss is defined as:
exp(simmi,, (t) ZxT)/)
Langte (F)) = log
(2)
2N 1= [[pa [mm)) exp(simm (ā., Tx)/T)
where m, E {0, 1} is the feature label, m, 0 denotes x; is normal feature and m. denotes T; is abnormal (mn has the same meaning). In RealIAD, each sample has ground-truth mask, we can downsample the mask to get feature labels. Further, we can analyze that the loss in Eq. only uses Ta as the positive pair. Other features with different labels from 23 are used as negative pairs. This ensures that we only perform contrast between normal and abnormal. In addition, we only utilize x; as the anchor feature, without using Zy as the anchor feature. That is to say, there will be no contrast between the augmented image and abnormal data in our angle-oriented contrastive loss.
Contrastive Loss. Unlike the above angle-oriented contrastive loss, this loss aims to enlarge the feature norm difference between normal and abnormal features. The basic idea follows nnee-llas-classifccall (OCC) learning [6], where normal features are optimized to be located inside the rrigie-centered hypersphere. In mathematies, feature norm can be viewed as the distance from the feature vector to the origin. Specifically, we use the pseudo- Huber distance / //|ar: || + 1 1 as the feature norm, which is a more robust distance measure that interpolates from quadratic to linear penalization 24 Then, we denote the distance from the feature I; to the hypersphere as d, = nn - r, where n, refers to the feature norm
[ 1 + I - 1, and r is the radius of the hypersphere and is
For symbol simplicity, we use N to represent the number of features in one feature level. Please note that we extract multi-level features, with contrastive learning on each level.
5

---

set to 0.4 (i.e., when ||: - 1|,, 1 0.4. Setting to 0.4 is equivalent to the unit hypersphere based on Euclidean distance). The ideal learning objective is to make all normal features contracted inside the hypersphere, and for features that are already within the hypersphere, we don't t need to excessively shrink towards the origin avoid mode collapse. To this end, we propose the following contraction loss:
Lcon (2:) = -logsig(-d; c
(3)
where logsig presents logarithmic sigmoid function. To better understand this loss, we further derive the gradients with respect the model weights (denoted as W) as follows:
HLoon (2:) = (1 11-a o1) ow logo, d: Öd,
Öd,
(4)
aw
OW
where o; presents sigmoid(-d, and can be explained as the probability that the feature inside the hypersphere. Thus, the features close to the origin have large probabilities (i.e. d: + -r, 0; + 1) and thus small o, and -logo,, while features outside the hypersphere have small Ø; + 0 and large -logo; (thus large gradient values). Thus, the loss in (3 can adaptively tune attention to assign larger gradients the features outside the hypersphere for better contraction.
For abnormal features, we expect that they are located outside the hypersphere and are highly discriminative from normal features. Thus, unlike OCC learning, contrastively optimize the norms of abnormal features to form a certain gap between them and normal features. We further introduce margin Ar and define an abnormal radius r' + Ar. When abnormal feature xy is inside the hypersphere with a radius r', it will be pushed outside of the hypersphere. In addition, to avoid overfitting anomalies during pretraining, don't further push away the abnormal features that already outside the hypersphere. Then the learning objective for æÿ is as follows:
logsig(- - n2)) c²
n2 Sr
Leoo (xrz)
0,
nJ
5 is based on Eq. but d, = r' n, is reversed (d; = n. MM:-7 r). We can analyze that the learning objectives in Eq. 3) and Eq 5 are encouraging to contract normal features inside the hypersphere with radius r and push abnormal features outside the hypersphere with radius r'. Thus, combining the two learning objectives, we call the final loss as norm-oriented contrastive loss, which is defined as:
fnorm (2x)) = [m.=0) Loon (x;) + [m,=1]] ('om(x))
In Eq. we unify the symbols, x; can be either normal or abnormal, indicated by m; = 0 or 1.
Total Loss. By combining angle- and norm-oriented contrastive losses, the whole loss for training is as follows:
2N liga Langte + Lnorm
2N X
Σ
X-
8=1
where A is set to 1 by default (please see the results in Tab 10 in Appendix E.
3.3 Feature Projector
As illustrated in the right part of Fig 2 the architecture of the Feature Projector is based on the Differently, we replace self-attention in vanilla Transformer with
popular Transformer network 43 our proposed Learnable Key/Value Attention. As we find that directly using vanilla Transformer as the Feature Projector has poor performance (see Tab
Learnable Key/Value Attention. Specifically, we randomly initialize N, learnable reference feature representations and define them as R E N. Then we utilize fully connected layer project R to the hidden dimension in Transformer into R, E NR and a single R, is used for all layers. Our idea for enhancing the vanilla self-attention is treat input features as Query vectors and learnable reference representations as Key and Value vectors, and then apply a cross-attention between them. After cross-attention, different from the original addition operation in the residual connection, we merge the input features and attention outputs through subtraction operation. We can understand the learnable key/value attention module in this way: the learnable reference representations can adaptively learn to represent normal patterns. As Value vectors, this makes attention outputs mainly contain normal patterns. By subtraction, we aim to eliminate the normal representations in the residual feature distribution adaptively learned by the network, further increasing the discrepancy between normal and abnormal residual features. See 4.3 Learnable Key/Value Attention based on the results in Tab b).
S.3 we provide more explanations to our

---

4 Experiments
4.1 Experimental Setup
Table 1: Anomaly detection and localization results with the original features and our pretrained features. ./. means image-level AUROC and PRO. i means our pretrained features are utilized in these AD models. For GLASS, use the GLASS-h variant in 6
12
25
DesnscNern
M4 10
116811
71 1/11
5013
HININ Rex
X
16.30 2911
3443
579-4
Un 2u0 oensa
02 5m9
<55 000 6-1-4
78 18831
914-11
079 1-30
54.00 011
48
02.02.1
05 013 1w6311.1
55-0713
1.4
04-0984
06 0E
60001 106
05-2904
16 3.20LL
M5C0
110077
552
7E
19
38.1403
945950
CLIP
1140
014712
06 0140:120 571.3100
1100-100 203 515.3
14-071L
02.5003
643 0013603
06 18.324
94 140011
628
P1481
550-1 190
10 10500
11:9700
0-3-1 1940
B1. 200
Inapited [1
014011
5:1 120
018-0
18
943
3351133
1000
118829
96.01 3851111
60 00.0041
000 204 2611
10.59LL
1.S.1.195..
05 05-992
46 333
008-12 098 8112
Datasets. We conduct extensive experiments on five AD datasets, including MVTecAD 2], VisA [56.. BTAD [5), MVTec3D 4 and MPDD [18], evaluate the effectiveness of our pretrained AD
representations.
Metrics. For image-level anomaly detection, the standard metric in anomaly detection, AUROC, is used [33 3 2). Nonetheless, since abnormal areas are usually smaller than normal areas, this may cause overestimated pixel-level AUROCs. This means that for small anomalies, even if they are not correctly located, the pixel-level AUROC is still high. Thus, we adopt the Per- Region- Ren--Reii-Ovvlaa (PRO) metric proposed in [3] for anomaly localization evaluation.
Implementation Details. Like most pretraining works, we employ multiple backbones for anomaly representation pretraining. In this way, we can more comprehensively validate our method and provide more pretrained AD network options. Specifically, we select strong modern pretrained backbones, including CLIP (29 series, DINOv2 [26] series and ImageBind [1I. Due to different network depths, the intermediate layers to output features are different among these selected models. The details are provided in Appendix D Then, for each layer, we construct Feature Projector (the number of layers is 1) for learning pretrained AD representations. We fix the parameters of the backbone networks, preserving their basic visual representation capabilities is beneficial (see (a)). We use Adam (28) optimizer with 1e learning rate to train. The batch size is set as 32 and the total training epochs are 10. The temperature hyperparameter and margin Ar are set as 0.15 and 0.75. The N2 is set to 2048. We use 42 as the random seed during pretraining. All training and test images are resized and cropped to 224 224.
Setup. Our work to learn proprietary pretrained features for AD tasks. Thus, we validate the effectiveness of our pretrained features by applying them into embedding AD methods to replace the original features. We mainly select representative methods as the baselines, including PaDiM 8 and PatchCore [33 (they are based on feature comparison without learnable parameters, thus can intuitively reflect the quality of pretrained features). Other methods include the influential CFLOW [12] and UniAD [50], and the latest state-of the-art method GLASS (6). When generating residual features, we match each input image with 8 reference samples. In addition, with the norm- oriented contrastive loss, a valuable advantage of our pretrained features is that normals and anomalies can be well distinguished based on feature norm. Then, we further propose a simple AD baseline (denoted by FeatureNorm), where we directly utilize feature norms as anomaly scores.
4.2
Main Results
We report the dataset-level average results across their respective data subsets in Tab To guarantee the rationality of result comparison, with the five backbones, we reproduce these methods based on their official open-source code and default hyperparameters and don't 't perform any hyperparameter tuning. By comparison, we can see that with our pretrained features, the performance of these
7

---

methods can be consistently improved on multiple datasets with various backbone networks, This demonstrates the superiority of our pretrained features and also confirms that learning proprietary pretrained features for AD tasks is effective and valuable. Especially, in some cases (e.g. CLIP- Base with PatchCore, DINOv2-Base with UniAD), the original features perform poorly, while our pretrained features can bring more significant improvements. Finally, shown in the last column (FeatureNorm') ) another superiority of our pretrained features lies in their capacity to achieve good AD performance based on simple feature norms even without any downstream AD modeling, while the ImageNet-pretrained features don't 't have such an advantage.
Explanations on Result Comparison, Our work doesn't t directly propose an AD model, but aims to provide better pretrained representations for AD tasks. Thus, applying our pretrained features in existing AD models to demonstrate performance improvement is the most reasonable way to validate the effectiveness of our work. Therefore, unlike previous AD papers that compare their proposed models with other mod- els, we focus on the performance improvement of the same model with and without our pretrained features.
Pretrained
W/ Pretrained
hora!l Aztormal
Hore ADTOITI
Figure 3: Feature t-SNE visualization. "w/o" and "w/" refer to without and with our pre- trained features. These features the "capsules" class of the VisA dataset. We show more visualization results in Fig 5 in Appendix F
Qualitative Results. Due to page limitation, the qualitative results are shown in Fig 4 in Appendix [F]
from
Intuitive Visualization. To intuitively illustrate the effectiveness of our method, show the t-SNE visu- alization of original features and our pretrained fea- tures in Fig 3 It can be found that with our anomaly representation pretraining, normal features are more compact, normal and abnormal features are more separated from each other. This intuitively demonstrates that our pretrained features are more discriminative AD representations.
4.3 Ablation Studies
In ablation studies, we utilize ImageBind |T]
the backbone network, followed by PaDiM and PatchCore as the baseline AD methods. Moreover, we also measure anomaly scores by feature norms (see the Setup part in Sec 4.1.. We perform ablation experiments on the VisA dataset.
Are pretrained features effective? As shown in aa) all three AD baselines can achieve better results based on our pretrained features (ExpID 2.2 and 2.7 vs. 2.1). This demonstrates that our anomaly representation pretraining method is effective.
Are residual features better pretrained AD representations? In Tab 2(a). the results show that residual features can bring further gains (ExpID 2.7 vs. 2.2), especially under FeatureNorm. In ResAD [4). the authors explained and verified that residual features are eaasaaeeerallzzble representations. Our studies also confirm the authors' statements and show that residual features are better pretrained AD representations compared to the vanilla features extracted by the backbone network.
Contrastive Losses. The ablation studies on contrastive losses are also in Tabī[a). Only learning with the angle-oriented or norm-oriented contrastive loss can achieve good performance (ExplD 2.5, 2.6 vs. 2.1), while combining the two contrastive losses can further outperform each single loss consistently (ExpID 2.7 vs. 2.5, 2.6). This indicates that the two proposed contrastive losses are complementary and simultancously optimizing the angle size and norm difference between features more conducive to obtain good representations for anomaly detection.
Does the backbone network need to remain fixed? In our method, we opt to keep the backbone network fixed and only optimize the Feature Projector. Because we think that the basic visual representation capabilities possessed in the pretrained backbone are still valuable and important, and training the whole feature network on AD datasets may cause impairment to the basic visual representation capabilities. The results in Tab verify our confirmation that the learnable backbone network will lead to performance degradation (ExpID 2.2 vs. 2.3 and ExpID 2.7 vs. 2.8). This also indicates that the AD field still needs larger-scale and better-quality datasets to support full- ball-backbone- based anomaly representation pretraining.
8

---

Table 2: Ablation study results. (a) Residual represents residual features. Non-residual means the vanilla features extracted by the backbone network. Angle and Norm mean angle- and normal-oriented contrastive losses, respectively. Fixed and Non-fixed represent whether the backbone network remains fixed during pretraining. Note that when the backbone network is fixed, the Feature Projector is required. Otherwise, there are no learnable parameters. (b) All Attentions actually represent the Transformer structure (Attention + MLP).
(a) Framework ablation studies.
(b) Architecture ablation studies for the Fea-
ture Projector.
ExHlD
Pootanal Rejresenatioos
Conineiiee
Feaurz Proeetor
Bockhore Newwar
PuD3M
PaudsCrve
FesturcNom
Losses
Fised
92.6366 3
916611.
Arciaactrr
P.DMM
PachCCrr
FatereNan
2.3
Nen-moiiaed
AzgledNNam
Piaed
95 530566
91.685.1
87.95119
MI Prosccse
66.6
682/791
0L3/903
2.3
Angle&Norn
Non-fiked
3003
91,91848
81.04332
Self Arcetion
05.7983 73.798.8
00 129884.3
00:011
Crrm Aneriain
10.1955
26
2.5
Revideal
Argle
Nuen
Foadd
96. 1088
05 93688 2
91.285.8
900064455
02.055.11
819951.
Leamable Kcy/lalaz Adcatiss
08.5/88.
19.7977
04.59133
2.7
Angle&Nono
Fooe
95 42967
94.697.0
94,249.0
Asgle&Nona
Nos-
$1,33560
78.432.1
Feature Projector Architecture. The results are shown in Tb) Our Learnable Key/Value Attention (LKV-Attt) can outperform other network architectures. In Cross Attention, we convert normal reference features into residual features and then utilize them as Key and Value. Compared to Self Attention, Cross Attention can perform better. This indicates that only embodying normal patterns in Key and Value should be effective. In our LKV-Attn, we further adopt learnable Key and Value. Compared to limited normal patterss in Cross Attention, the learnable reference (Sec(3.3) in our LKV-Attn can adaptively learn to represent normal patterns and better cover key normal feature patterns in the residual feature distribution during pretraining.
For hyperparameter ablation studies, please see Appendix
4.4 Further Analysis
Sample Efficiency. We further conduct experiments about sample efficiency, where we utilize only 10% normal samples from each downstream AD dataset for training. The results are in Tab Ta). Compared to the results in Tab with less training data, our pretrained features can bring more significant performance improvement, indicating that our pretrained features are beneficial for improving the sample efficiency of downstream AD models.
Table 3: Sample efficiency and robustness analysis. t means our pretrained features are utilized in these AD models.
(a) Sample efficiency experiment results.
(b) Robustness experiment results.
Dutasets
PaDIM
PaboM
PanchCore
PaschCore
Datasets
PaDIM
PaDxM
PuchCCore
PrxchCore
MVTecAD
$1.4/88.6
06.59862
98.211.087.7
MVTscAD
06.6/5277
95.2-1
87.7/90.6
892/0.581.4113
VisA
88.977.77
93.9 13.9-50085.7
8237556
9774-36799-4-3
96.9:35/77.71
7-02
80.3/5.2
9-3.4
BTAD
90.55
177.4-25
85.162.44
MVTec3D
66,385.8
74.3964.7
30.8-5.5977.1
MVReCD
72.055.1
71.0886
73.0-2.000.5- 1.9
MPDD
63.1755.
19:2
74.1/6444
MPDD
793853
84.0-1.200
83.00.1
84.5-1 1/85 8-0.
Robustness to Noise. We further conduct experiments with noisy data to investigate the robustness. Specifically, we add abnormal data from the test set to the training set with noise ratio of 0. We adopt the "overlap" setting from SoftPatch 19]. The results are in TbbB]b). It can be found that the performance of AD models significantly decreases when there is noisy data. However, our pretrained features still bring performance improvement, and the magnitude is even greater compared to the results in Tab This indicates that our pretrained features original features. For the robustness, we think that a possible explanation may be: Although some noises are added the training set, it is still dominated by normal features. For PatchCore as an example, the training process is to subsample a coreset. With better representation, abnormal features more likely be sparse outliers, making them more likely not to be sampled into the coreset. Thus, the results show better robustness.
more robust to noisy data than the
Reference Set Sensitivity. Since residual features depend few-shot normal refer- ence samples, different normal reference samples may result in performance variations. Thus, we further investigate the sensitivity of our pretrained features to the reference set.

---

We randomly sample three reference sets (with shot), conduct experiments respectively. We report variance bars in Tab Overall, our pre- trained features are not very sensitive to the refer- ence set, and the performance is not significantly affected by reference variations. However, we further point out that if the reference set is not representative enough (lacking some normal patterns), the results would be more affected. We provide more discussions in Appendix [annd introduce feasible method for tackling this issue.
Table 4: Reference set sensitivity experiments.
Dultests
Pub3
PabiM
PaideCln
PaccCuue!
MVTrcAD
47 9
98.6
1482 1-005
98 5188 9
985-0000 4+0. +0.39
VisA
42,636.3
054-n 2188-277777-01 7-009
914481. 9
944-0 2-10 57
BEAD
92.6759 9
959:0:296600001
94.656.7
956:0000000000 Da L0
MVTec30
79.5903
78-886.3
82 (30 2947333333
MPDD
91.0092.0
958::046955000000
926/99.1
Comparison to Fine-tuning Baselines. In Appendix B.1 we further discuss and compare with previous per-dataset finettuning methods, such as FYD 54 MSC [31] and PANDA [30].
Pretraining-test Data Leakage. A non-negligible concern about work is whether there may be anomaly data leakage between the pretraining dataset and the test datasets. We discuss this issue in Appendix B.2
4.5 Few Shot Anomaly Detection
As stated in the Setup part in Sec 4.c and shown in T one valuable advantage of our pre- that the feature norms can
Table 5: Performance comparison with FSAD methods the MVTecAD and VisA datasets. "x" indicates results of these methods are from Win- CLIP [17]. "#" indicates results of these methods are from KAG-Prompt (38) "I" indicates these methods employ ImageBind tractor.
trained features be directly used as anomaly scores. This means that when only few-shot normal samples are accessible, we can easily construct few-shot AD (FSAD) method (denoted by FeatureNorm). Specifically, for an input sample, we extract pre- trained residual features. Then, for a pretrained feature 24, the L2 norm || ||2 is used as the anomaly score. To demonstrate the potential of our pretrained features for FSAD, compare our FeatureNorm with mainstream FSAD meth- ods. We follow the 2-shot and 4-shot settings in KAG-Prompt (38, the results are in Tab Com- pared with these competing methods, our simple FeatureNorm is comparable and even superior (on VisA). This demonstrates that the essence of FSAD still lies in the representation ability of features. With better AD representation features, we can achieve good FSAD results without the need to design elaborate methods (e.g., the Kernel- Aware Hierarchical Graph in KAG-Prompt). Compared to the most relevant work, ResAD, our simple FeatureNorm can outperform it, especially on the VisA dataset. Moreover, a major advantage of our work that we can provide better representation features for downstream AD methods, while ResAl) cannot empower other AD models.
the feature ex-
Seay
View
35.1
41.3
813
2.6
65.5
11.
8113
38.6
66.3
PECERAD
$5.2
$8.5
949
RacAD"
544
156
54.3
SPADE"
05h301
34.5
$70
54.5
15.5
193
174
05.6
665
015
$4.1
CV78304
Feidd"
582
469
52.8
5 Conclusion
In this paper, we explore the problem of anomaly presentation pretraining, which is important for the anomaly detection field but has been inadvertently overlooked. We propose novel AD representation learning framework, which learns pretrained AD representations based on residual features and consists of angle- and norm-oriented contrastive losses fully optimize feature discrepancies. Experiments comprehensively demonstrate that our proprietary pretrained features consistently surpass the original prompted the prosperity of computer vision. Analogously, in the AD field, valuable question what kind of representations can serve as the fundamental (general) representation for anomaly detection. Furthermore, more attention could be paid to anomaly representation pretraining in future work, rather than constantly focusing on designing more sophisticated AD models dependent on ImageNet-pretraing networks. The limitations of our method are discussed in Appendix B.5
features. As we all know, pretraining on ImageNet has
10

---

Acknowledgments
This work was supported part by the National Natural Science Fund of China (62371295) and the Science and Technology Commission of Shanghai Municipality (22D72229005).
References
[1] Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. arXiv preprint arXiv: 2002. 10445, 2020.
[2] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad compre- hensive real-world dataset for unsupervised anomaly detection. In CVPR, 2019.
[3] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In CVPR, 2020.
[4] Paul Bergmann, Xin Jin, David Sattlegger, and Carsten Steger. The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. arXiv preprint
2021.
[5] Paul Bergmann, Sindy Lowe, Michael Fauser, David Sattlegger, and Carsten Steger. Improv- ing unsupervised defect segmentation by applying structural similarity to autoencoders. In International Conference on Computational Vision Technologies and Applications, 2019.
[6] Qiyu Chen, Huiyuan Luo, Chengkan LVV, and Zhengtao Zhang. A unified anomaly synthesis strategy with gradint ascent for industrial anomaly detection and localization. In ECCV, 2024.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[8] Thomas Defard, Alcksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: a patch distribution modeling framework for anomaly detection and localization. In 1st International Workshop on Industrial Machine Learning, 2021.
(9] Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embed- ding. In CVPR, 2022.
[10] Matic Fučka, Vitjan Zavrtanik, and Danijel Skočaj. Transfusion a transparency- diffusion model for anomaly detection. In ECCV, 2024.
[11] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
[12] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In IEEE Winter Confer- ence on Application of Computer Vision, 2022.
[13] Jianfei Hu Hanxi Li, Bo Li, Hao Chen, Yongbin Zheng, and Chunhua Shen. Target before shooting: Accurate anomaly detection and localization under one millisecond via cascade patch retrieval. arXiv preprint arXiv: 2308.06748. 2023.
[14] Haoyang He, Jiangning Zhang, Hongxu Chen, Xuhai Chen, Zhishan Li, Xu Chen, Yabiao Wang, Chengjie Wang, and Lei Xie. Diad: A diffusion-based framework for multi-class anomaly detection. In AAAI, 2024.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[16] Liren He, Zhengkai Jiang, Jinlong Peng, Wenbing Zhu, Liang Liu, Qiangang Du, Xiaohib Hu, Mingmin Chi, Yabiao Wang, and Chengjie Wang. Learning unified reference representation for unsupervised multi-class anomaly detection. In ECCV, 2024.
11

---

[17] Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero-/few-shot anomaly classification and segmentation. In CVPR, 2023.
[18] Stepan Jezek, Martin Jonak, Radim Burget, Pavel Dvorak, and Milos Skotak. Deep learning- based defect detection of metal parts: evaluating current methods in complex conditions. In 13th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops, 2021.
[19] Xi Jiang, Jianlin Liu Wang, Qian Nie, Kai Wu, Yong Liu, Chengjie Wang. and Feng Zheng. Softpatch: Unsupervised anomaly detection with noisy data. In NeurlPS, 2022.
[20] Daehyun Kim, Sungyong Baik, and Tae Hyun Kim. Sanflow: Semantic- normalizing flow for anomaly detection and localization. In NeurlPS, 2023.
[21] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu- tions. In Conference and Workshop on Neural Information Processing Systems, 2019.
[22] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Grrabsa-Barwinng Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
[23] Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. Simplenet: A simple network for image anomaly detection and localization. In CVPR, 2023.
[24] Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Muller. Explainable deep one-class classification. In International Conference Learning Representationss, 2021.
[25] Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. Vt-adl: A vision transformer network for image anomaly detection and localization. arXiv preprint 10Xx:22104.10036, 2021.
[26] Maxime Oquab, Timothee Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. In Trans. Mach. Learn. Res., 2024.
[27] Pramuditha Perera and Vishal M. Patel. Learning deep features for one-class classification. IEEE Transactions on Image Processing, 2019.
[28] Diederik P.Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[30] Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen, Panda: Adapting pretrained features for anomaly detection and segmentation. In CVPR, 2021.
[31] Tal Reiss and Yedid Hoshen. Mcan-shifted contrastive loss for anomaly detection. arXiv preprint arXiv:2106.0086 1344444446 2021.
[32] Oliver Rippel, Arnav Chavan, Chucai Lei, and Dorit Merhof. Transfer learning gaussian anomaly detection by fine-tuning representations. In Proceedings of the 2nd International Conference Image Processing and Vision Engineering, 2022.
[33] Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Scholkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In CVPR, 2022.
12

---

[34] Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semi- supervised defect detection with normalizing flows. In IEEE Winter Conference on Application of Computer Vision 2021.
[35] Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Fully convolutional cross- cross-scale-flows for image- defect detection. In IEEE Winter Conference on Application of Computer Vision 2022.
[36] Lukas Ruff, Robert A. Vandermeulen, Nico Gornitz, Lucas Deecke, and Shoaib A. Siddiqui. Deep one-class classification. In International Conference on Machine Learning, 2021.
[37] Mohammadreza Salehi, Niousha Sadjadi, Soroos Hossein Rohban, and Hamid R.Rabiee. Mul- tiresolution knowledge distillation for anomaly detection. In CVPR, 2021.
[38] Fenfang Tao, Guo-Sen Xie, Fang Zhao, and Xiangbo Shu. Kernal-aware graph prompt learning for few-shot anomaly detection. In AAAI, 2025.
[39] Xina Tao, Xinyi Gong, Xin Zhang, Shaohua Yan, and Chandranath Adak. Deep learning for unsupervised anomaly localization in industrial images: A survey. IEEE Transactions Instrumentation and Measurement, 2022.
[40] David MJJ. Tax and Robert P.W. Duin. Support vector data description. In Machine Learning. pages 45 45-66,22 2004.
[41] Tran Dinh Tien, Anh Tuan Nguyen, Nguyen Hoang Tran, Ta Duc Huy, Soan T.M Duong, Chanh D. Tr. Nguyen, and Steven Q.H. Truong. Revisting reverse distillation for anomaly detection. In CVPR, 2023.
[42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.066 07.047788..... 2018
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[44] Chengjie Wang, Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jiangning Zhang, Zhihao Gu, Shuguang Qian, Mingang Chen, and Lizhuang Ma. Real-iad: A real-world multi-view dataset for benchmarking versatile industrial anomaly detection. In CVPR, 2024.
[45] Jie Yang, Yong Shi, and Zhiquan Qi. Dfr: Deep feature reconstruction for unsupervised anomaly segmentation. arXiv preprint arXiv: 2012.07122, 2020.
[46] Xincheng Yao, Zixin Chen, Guangtao Zhai, and Chongyang Zhang. Resad: A simple framework for class generalizable anomaly detection. In neurIPS, 2024.
[47] Xincheng Yao, Ruoqi Li, Jing Zhang, Jun Sun, and Chongyang Zhang. Explicit boundary guided semi-push-pul contrastive learning for supervised anomaly detection. In CVPR, 2023.
[48] Xincheng Yao, Chongyang Zhang, Ruoqi Li, Jun Sun, and Zhenyu Liu. One-for-all: Proposal masked cross-class anomaly detection. In AAAI, 2023.
[49] Jihun Yi and Sungroh Yoon. Patch svdd: Patch-level svdd for anomaly detection and segmenta- tion. In Asian Conference on Computer Vision, 2021.
[50] Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. arXiv preprint arXiv:2206.0366 2022.
[51] Vitjan Zavrtanik, Matej Kristan, and Danijel Skocaj. Draem: A discriminatively trained reconstruction embedding for surface anomaly detection. In ICCV, 2021.
[52] Xinyi Zhang, Naiqi Li, Jiawei Li, Tao Dai, Yong Jiang, and Shutao Xia. Unsupervised surface anomaly detection with diffusion probabilities model. In ICCV, 2023.
[53] Xuan Zhang, Shiyu Li, Xi Li, Ping Huang, Jiulong Shan, and Ting Chen. Destseg: Segmentation guided denoising student-teacher for anomaly detection. In CVPR, 2023.
13

---

[54] Ye Zheng, Xiang Wang, Rui Deng, Tianpeng Bao, Rui Zhao, and Liwei Wu. Focus your distribution: Coarse-to-fine non-contrastive learning for anomaly detection and localization. In ICME, 2022.
[55] JiaWen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learning with few-shot sample prompts. In CVPR, 2024.
[56] Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the- difference self-supervised pre- mre-trainnng for anomaly detection and segmentation. In ECCV, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have summarized contributions well in the abstract and introduction, and the method and experiments sections also reflect these contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix B.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please the implementation details in Sec 4.1 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: As mentioned in Abstract, the open-source code will be available at https gegithub.com/xcyyee wwtp:////cyaoo/////eetaaaala ADPretrain
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
gwwcc//ggges//eee hwt////ooeisssiiooooco seiisssssoyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please see Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Due to too large amount of experiments (please see Tab I with six methods, five AD datasets, five backbones, and experiments with/without our pretrained features, 300 experiments), it's very time-consuming
we need to conduct total x to use different random seeds for multiple experiments. To this end, we ensured that all experiments were conducted under the same condition with the random seed set to 42. In addition, in Tab utilize multiple random seeds to randomly sample different reference sample sets and report the error bars based on partial AD methods. These results demonstrate that our pretrained features
very sensitive to the reference set.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the computation costs in Tab
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte::////euiip// https////b...////cccccieeeelllll
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics, and believe that our research conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix B.7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We think that our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets used in the paper all open-sourced, and we have also cited corresponding papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper proposes novel anomaly representation learning framework, does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in our work does not involve L.MSs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
A Significance of Anomaly Representation Pretraining
Strictly speaking, from the implementation perspective, our method is not true "pretraining", as it didn't achieve pretraining the whole backbone network. However, the usage of "pretraining" has two reasons: 1) We are indeed following the basic paradigm of pretraining: learning on a large-scale dataset, and the learned features can be transferred to downstream datasets and models. 2) More importantly, we want to emphasize that in the current development stage of anomaly detection, fundamental representations specifically learned for anomaly detection are of great significance. Current mainstream AD models mostly pretrained networks to extract features. However, regardless of supervised or self-supervised pretraining on natural images, the pretraining process does not match the goal of anomaly detection. From the perspective of the development prospects of AD, it's necessary for AD tasks to have specialized pretrained features instead of continuously using basic pretrained features. Our work is only an early exploration, and we hope to inspire more future works to focus on anomaly representation pretraining.
B
More Discussions
B.1 Discussions with Fine-tuning Baselines
We further provide discussion with the previous finetune-based baseline methods, such FYD [54) MSC [3I, and PANDA (30]. These methods aim to finetune pretrained features to adapt to the target distribution in specific AD datasets. As the FYD paper does not provide open-source code, we can only compare with the results provided in the paper. In FYD, anomaly detection achieved based on PaDiM. The paper provides the results MVTecAD, which are 97.7/98.2 mamage-evee/ppixellleed AUROCs). Based on PaDiM, the results of our method are 98. 1/9..5. In PANDA and mean- shift contrastive loss (MSC), feature adaptation is based on global features, so only image-level anomaly detection is performed. The two papers provide image-level AUROC on MVT with values of 86.5 and 87.2. For more reasonable comparison, we utilized the open-source code of MSC and trained an adapted network (based on WideResNNe10)) on MVTecAD. Then, we utilized PaDiM for AD modeling, and the obtained results are 96.8/97.0. 00 Thus, compared with PANDA. MSC, and FYD, our method has advantages in providing better features for AD models. Moreover, adaptive feature fine-tuning requires specialized fine-tuning on each dataset. By comparison, more important advantage of anomaly representation pretraining is that pretrained features can be used on multiple downstream AD datasets without the need for fine-tuning on each dataset.
is crucial to point out that FYD does not utilize large-scale AD datasets for pretraining when improving feature representations. Thus, what impact does the dataset scale have feature fine- tuning? To answer this, we further perform pretraining on the MVTecAD dataset. The results are in T
Table 6: Experimental analysis of the impact of pretraining dataset scale.
Datasets
PaDiM (RealIAD)
PaDiM MVTecDD)
PaschCore (RcalLAD)
PatchCore (MYIecAD)
MVTxcAD
98.3/92.1
97.392.11
98. 9/88.88
98.8/83.0
VisA
95.688866
93.088.6
94 84886661 3
93.4/56
BTAD
95.9/76.8
95.6/76.6
95.6/67.3
94.3/665
MVTxc3
84.4/92.0
82.8/91.2
82.6/700
80.687.3
MPDD
96,4/95.1
92.9/93.5
94.8/94.2
93,1/93.4
Impact of Pretraining Dataset Scale. It can be seen that with pretraining on MVTecAD, the results are overall lower than the results obtained based on RealIAD pretraining. Thus, we think that the scale of the dataset will have an impact, a larger scale is more advantageous for pretraining (better transferability of the learned features). This also reflects that the AD field still needs larger and qughet-quality datasets in the future to better support anomaly representation pretraining.
However, in FYD, the transferability of the fine-tuned features is not discussed. FYD, after fine-tuning on MVTecAD, the fine-tuned model is effective on MVTecAD, but it has not been verified whether it is still effective on other datasets. We think that FYD requires specialized fine-tuning on each dataset. The reason is that the fine-tuning process in FYD is mainly to accomplish spatial
22

---

alignment (beneficial to the PaDiM method) on feature maps, As the spatial distributions of objects in different datasets are different, the learned affine transformation parameters one dataset should be hard to apply to other datasets. Therefore, even if we employ FYD to fine-tune on ReallAD, the adapted network cannot be directly applied to other downstream datasets. Nevertheless, in our paper, the results in T have comprehensively verified that the pretrained features have good transferability in downstream AD datasets.
B.2 Discussions on Data Leakage
A concern about our work is whether there may be anomaly data leakage between the pretraining dataset and the test datasets. We think that the data leakage issue does not need to be concerned. Because the product categories in RealIAD are completely different from those in the test AD datasets, and the anomaly types are also quite different. Even there are some similar types of anomalies (e.g., broken), the visual appearance displayed on different products will also be different. For data leakage, it's hard to calculate a numerical indicator. Due to different products, the similarities between pretraining images and test images are all low, making it hard to distinguish what constitutes anomaly data leakage. In Tabl6 we report results of both pretraining and testing on the MVTecAD dataset. This is a serious data leakage, but compared to RealIAD pretraining, it doesn't 't bring significant performance improvement. This indicates that data leakage is not a critical issue for the performance gains in work.
B.3 Discussions with SPD
We further provide a discussion with a previous pretraining method, SPD [56]. In the paper ppot-the-difference (SPD), the authors propose data augmentation strategy called SmoothBlend to create slight local differences (or called pseudo-anomalies) Then, the authors create pseudo- anomalies natural images from ImageNet use as negative samples in contrastive learning. The model is pretrained on ImageNet, while our model is pretrained on a real AD dataset, RealIAD. (1) Compared pseudo-anomalies, the negative samples used in our method are more realistic and more in line with the abnormal characteristics in real-world scenarios. (2) In SPD, similar to classical SSL methods (SimCLR, MoCo), contrastive loss is constructed based on global features, while we construct contrastive loss based on more fine-grad aineggrinee local features. Although SPD mentioned that can promote local sensitivity compared SimCLR, MoCo, etc., still may lead the loss of information about local pseudo-anomalies in the global features, as the local augmentation is slight. For fine-graiee ine-geai task like anomaly detection, contrastive learning based on fine- ane-grained local features should be more suitable. (3) SPD was evaluated only on the ResNet-50, PaDiM, and VisA and MV datasets, while we evaluate our method on more backbones, AD models, and AD datasets.
B.4 More Discussions with ResAD
We think that we should further discuss some similarities and differences between our work and ResAD (46.. Compared to ResAD, we mainly employ the wonderful residual features proposed in ResAD as our pretrained AD representations. Because we expect that pretrained features can serve as general features in anomaly detection, namely, it's best for pretrained representations to be domain-invariant. As the authors explained in [46), the residual features can be regarded as class- generalizable representations, which satisfy our expectations just right. The results in Tab also demonstrate that residual features are better pretrained AD representations compared to the vanilla features extracted by the backbone network. Our studies also further confirm the authors' opinions in [46] that residual features have the potential to be general representations in anomaly detection. However, our work and ResAD are clearly two different works, with differences in motivations, concemed problems, and implementation methods.
(1) Different Motivations and Concerned Problems. ResAD explores the problem of class- generalizable anomaly detection. The authors mainly aim to learn generalizable AD model that can be directly applied to new classes to accomplish anomaly detection. By comparison, our work focuses on very valuable but inadvertently overlooked problem: anomaly representation pretraining. We aim to learn better and proprietary representation features for AD tasks through contrastive learning pretraining on large-scale AD datasets.
23

---

(2) Tlass-eeeralliable Anomaly Defection vs. Anomaly Representation Pretraining, Although class- geassseneralizationn anomaly detection can also be seen as pretraining an AD model on dataset and then applying it to new datasets. However, it's still significantly different from anomaly representation pretraining. as anomaly representation pretraining aims to provide better AD representation features for downstream AD methods, while class- geneesegeeall anomaly detection (ResAD, InCTRL. (55)) cannot empower other AD models. This is why in the experiments, we didn't perform reverse setting (train on the combination of MVTecAD, VisA and other datasets and eval on RealLAD). We need to validate the effectiveness of our pretrained features in existing AD methods, rather than verifying the cross-class generalization ability of AD models. The models in Tab T are also first trained on normal samples with our pretrained features, and then evaluated.
(3) Different Implementation Methods. Our work doesn't it construct a model that can be directly used for anomaly detection, while providing pretrained representation features for AD tasks. Our pretraining framework is based on contrastive learning. To fully optimize discrepancies between normal and abnormal features, we propose angle- and norm-oriented contrastive losses from the feature similarity perspective. ResAD constructs caas-generallizble AD model that accomplishes anomaly detection based on normalizing flow model [21] to learn residual feature distribution. In addition, ResAD uses fixed normal samples as reference to generate residual features, without considering the semantic misalignment issue. We realize this issue and provide an improvement (the semantic-aligner reference matching module, Appendix C) for generating semantic-aligned residual features.
(4) As for performance, we compare the FSAD performance with ResAD. As shown in Tab our simple FeatureNorm is better than ResAD, especially on the VisA dataset. Moreover, compared to the more sophisticated normalizing flow modeling in ResAD, calculating feature norms in our method is quite simple. This also demonstrates the value of our pretrained features.
B.5 Limitations
In this work, unlike conventional AD research works, we explore a very valuable but inadvertently overlooked problem: anomaly representation pretraining. We propose novel AD representation learning framework to learn proprietary pretrained features for AD tasks. Even if incorporating our pretrained features into embedding- based AD methods can manifest good performance improvement on five AD datasets and five backbones (Tab[). there are still some limitations of our work.
One limitation of our work is that we fix the backbone network and only optimize the Feature Projector during training. However, we think that for an ideal feature extraction network in AD tasks, the backbone part should be specially designed based on the characteristics of anomaly detection and effectively learned during pretraining. For example, when encoding one image, the backbone network can simultaneously consider multiple normal reference samples to increase the normal context patterns. Then, the contrast between normal and abnormal can be implicitly embodied in the network encoding process. Thus, the encoded normal and abnormal features would be more discriminative.
Another limitation is that our pretrained features can only be incorporated into embedding- based AD methods. Other AD methods that are not based on pretrained feature extractors (e.g., cestusion-basedd cannot benefit from our method. Therefore, future work should also focus on exploring pretraining frameworks for more AD methods.
B.6 Computational Cost
In this work, we accomplish anomaly detection based on some embedding- based AD methods. The computational cost of a complete AD model is determined by both the feature extractor and the subsequent AD method. Compared to using SaageNet-pretrained features, our method mainly introduces extra cost in the Feature Projector. For different backbones, the computational costs of the Feature Projector are different. With the image size fixed as 224 x 224, we calculate the number of parameters and computation FLOPs in the Feature Projector. The results are shown in Tab 7
24

---

Table 7: Computational cost of the Feature Projector with different backbone networks.
DINOv2-Base
DINOv2-Lage
CLIP-Base
CLIP-Large
ImageBind
parameters
21.3M 137.7G
37.8M 309.2G
21.3M 137.7G
37.8M
FLOPs
309.2G
483.2G
B.7 Social Impacts and Ethics
As work for learning robust and discriminative pretrained representations for anomaly detection, the proposed method does not suffer from particular ethical concerns or negative social impacts. All datasets used are public. All qualitative visualizations are based on industrial product images, which don't 'o infringe personal privacy.
C Semantic-Aligned Reference Matching
Discussions on Reference-Set Sensitivity. Residual features are mainly relied on mutually eliminat- ing class-related normal patterns in features through subtraction (please see the ResAD paper [6 for more explanations), for being class class-generes generalizablee. However, if normal images have significant variability,, it may cause the few-shot reference samples to be not representative enough (lacking some class-related normal patterns). This may lead to certain class-related normal patterns in the input feature can't 't be effectively eliminated. Then, the discriminability of residual features under few-shot settings will be compromised.
For practical applications, this issue should be particularly focused and reasonably addressed. Of course, the simplest solution to increase the number reference samples. This is feasible, as in practical applications, the number of reference samples is usually not as strict as the 8-shot. However, in practical applications, we expect that the reference samples can fully represent their class, so it's best to have sufficient differences between the reference samples. Thus, the sample selection strategy cannot be random. A feasible method is to first cluster all available normal samples into different clusters based samples, we evenly distribute it to each cluster. When selecting from a cluster, we can prioritize selecting samples closer to the center. Another feasible strategy is to match spatially aligned samples as reference samples for each input sample. Below, we provide a corresponding approach.
clustering algorithm (c.g., KMeans). Then, based on the number of reference
Residual Features with Semantic-Aligned Reference Matching. When constructing residual features, it's required to first match the nearest feature from the reference feature bank. To maintain the efficiency of residual feature generation, the reference feature bank cannot be too large. Thus, in [46], the authors use fixed few-shot normal samples to construct the reference feature bank. Although efficiency is guaranteed, it may sacrifice the representation ability of residual features. In real- world scenarios, diffcrent normal samples of the same class may have semantic differences (e.g., variable object components shapes, misaligned objects). When the semantic differences among normal samples are too large, it can cause fixed few-shot normal samples can't 't cover all normal patterns, thereby the reference feature bank may not be representative enough. To address this, we can select proper camantic-aligned (i.e., the components, shape, and orientation of the object in two images are relatively consistent) reference samples for each input image. In specific, we propose a lightweight semantic-aligned reference matching method, which calculates statistical histograms (distances from local features clustering centers) for the image and then measures the global distance between two images by the KI. divergence between histograms. Two aligned images usually have close spatial feature distribution, so the histograms are similar, namely, smaller KL divergence. As histograms are low-dimensional, the calculation of KL divergences has a very low cost. Thus, the matching module will not introduce too much cost. Finally, we indicate that this module not used during pretraining, while it can be used when we apply pretrained features to downstream AD datasets.
comantic-aligned reference matching method that can retrieve semantic-aligned samples from the normal sample set for the input image Z. Given total N normal images, extract first layer features with pretrained lightweight network (e.g., ResNet-18), 8), the corresponding features are denoted as { F, E BR X xCi Then, for each image, the feature map F. is evenly divided into S S grids
Semantic-Aligee
Reference Matching. We describe
1,2,..., N..
F.
0.
S
xC1
U.
..2....,
(8)
25

---

For each grid, we collect all raw patch features of different normal images together into the feature set .. NH, Wi /S The K-means clustering algorithm performed on p"." obtain Ne clustering centers C {ck BCIk k 1, 1,2..... Ne). For each grid, we can obtain Ne clustering centers, and then the total number of centers is Ne.
pp.." = (Flatten(F, "* |
The feature map F; and clustering centers {C E RRNe Ju, 1........ S) are further utilized to calculate the block-wise statistics for each normal image 1, Specifically, in the (u, v)-th grid, we calculate the cosine similarity between the grid feature map F." IR " x " xO; and all centers C E R Ncx Then we obtain H1 /52 histograms denoted by H. = [hx E RRSNN Nolk = .,2. H1 H.WisS ) where hgg denotes the cosine distance histogram of k-thh feature in the grid with respect to the codebook C. The histograms are then normalized so that ||/x/| = 1, vk. Then, for the input image and the i-th normal image 4, we can calculate the KL divergence between their histograms to measure the spatial alignment degree between them. Specifically, the calculation of KL divergence is based on grid, we calculate the block-wise KL divergence D LL ,H")) for each block (u, 2), where H means the histograms in (u, v)-th grid of the input image. As in one grid, there are H₁ W histograms, the shape of D. should be R Ws If the two grids are semantically similar, the maximum KL divergence needs to be small enough. Therefore, we only select the maximum KL divergence df from D as semantic alignment degree between two grids, d. = (.." The global semantic alignment degree between and I; is estimated as:
8
S
Dalign
M.U
Based on the global semantic alignment degree, we can sort the global distances between the input image and N normal images and then retrieve the top-K neighbor normal images (we denote the set of indices as 1) few-shot normal reference images Nref = -.,,,,, l.g...,, L.x J. In implementation, the hyperparamters S, Ne. and K are set as 5, 5, and 8, respectivcly.
reference matching approach is to provide proper semantic-aligne reference samples for each input image. Generally speaking, such reference samples should be more reference-informativ especially when the test images are highly non-aligned. Moreover, in practical usage, when inputting a test image, the semantic-aligned reference matching approach will not introduce too much extra cost for the inference process. We can employ it as a preprocessing step by calculating histograms for all collected normal samples and storing these histograms. Then, the input image only needs to calculate its own histograms and calculate the block-wise KL divergences with the stored histograms, and the calculation of KL divergences has a very low cost. When using ResNet-18 18 as the lightweight network, the cost of calculating the global semantic alignment degree between / and I; is only about 0.1 GFLOPs.
Further Discussion. Employing the semantic-alignes
D Implementation Details
Like most pretraining works, we employ multiple backbones with different architectures and parame- ter scales for anomaly representation pretraining. Specifically, we select DINOv2-B (26). DINOv2-L (26). CLIP-B (29). CLIP-L [29]. and ImageBind [II] as the backbone network. Due to different net- work depths, the intermediate layers to output features are different among these models. DINOv2-B and CLIP-B have total of 12 layers, and we use the [3. 6, 9, 12] layers to output features. Network layers in DINOv2-L and CLIP-L are 24, and features from the [6, 12, 18, 24] layers are used for training. For ImageBind, the outputs from the [8, 16, 24, 32] layers are used for training.
Implementation Details About the Results in Tab In Tab to guarantee the rationality of result comparison, with the five backhones, we reproduce these methods based on their official open-source code. For each method, we use the default hyperparameters and don't 't perform any hyperparameter tuning. When applying our pretrained features in these methods, we make sure that we only replace the original features without any additional modifications.
E Hyperparameter Sensitivity
In this section, we conduct more ablation experiments on the main hyperparameters our method to illustrate their sensitivity. Following the main text, in hyperparameter ablation studies, we also utilize
26

---

ImageBind as the backbone network and produce results with PaDiM, PatchCore, and FeatureNorm as the baseline AD methods.
Temperature Hyperparamter т. In Tab 8 we ablate the temperature hyperparameter T in the angle-oriented contrastive loss (see Eq. 22) The results show that larger t (i.e. 0.3) will lead to degraded results. When T is set to 0.3, the results produced by FeatureNorm are the best, but the PRO result under PatchCore is poor. When is set to 0.1, the results produced by Patchcore are the best, but the results under FeatureNorm are poor. Therefore, by overall consideration, setting the temperature hyperparameter 0.15 is the most suitable choice.
Table Ablation studies about the temperature hyperparameter T in the angle-oriented contrastive loss (see (2. / means image-level AUROC and PRO.
0.15
0.2
0.25
0.3
0.5
Method
PaDiM
95.4/87.8
95.5/88.2
95.4/87.9
95.3/87.8
95.1/87.5
94.9/87.3
PatchCore
94.4/87.5
94.6/87.0
94.3/87.4
94.5/86.5
94.5/86.5
94.6/85.5
FeatureNorm
93.3/87.7
93.9/88.1
93.9/87.9
94.1/87.9
94.1/88.1
93.8/87.5
Margin Hyperparamter Ar. In T we ablate the margin hyperparameter Ar in the norm-oriented contrastive loss (see 5)). The results show that our method is not very sensitive to the margin Ar. We think the reason should be that in the norm-oriented contrastive loss, we only need set certain margin to ensure that the norm of normal and abnormal features can be distinguished, rather than having to set a large enough margin. When Ar set to 0.75, the results under three baseline methods are overall better compared to other Ar values. Therefore, we decide to set the margin hyperparameter Ar to 0.75 our method.
Table 9: Ablation studies about the margin hyperparameter Ar in the norm-oriented contrastive loss (see Eq. 5) Ar).
Ar
0.1
0.2
0.3
0.5
0.75
1.0
Method
PaDiM
95.5788.2
95.5/88.3
95.5/88.1
95.5/88.1
95.5/88.1
95.5/88.0
PatchCore
94.6/87.0
94.6/87.2
94.6/86.6
94.6/87.4
94.6/87.5
94.3/87.2
FcatureNorm
93.9/88.1
93.9/88.0
93.9/88.0
93.9/88.0
94.0/88.2
94.0/88.4
Loss Weighting Coefficient X. The results are shown in Tab I0 It can be found that with À set the best results can be achieved.
Table 10: Ablation studies about the loss weight À in the total loss (see Eq. 7
0.5
1.5
2
3
5
Method
PaDiM
95.4/88.0
95.5/88.1
95.5/88.0
95.4/87.7
95.2/87.2 95.3/87.5
PatchCore
94.3/87.8
94.6/87.5
94.5/87.0
94.6/87.3
94.6/87.6
94.5/87.2
FeatureNorm
93.5/88.0
94.0/88.2
93.9/88.1
93.9/87.9
93.9/88.1
94.1/88.0
Number of Layers in the Feature Projector L.. The results are shown in Tab 11 The results indicate that more network layers don't 't bring obvious performance improvement, instead, the PRO results under PatchCore are lower than the counterpart (87.5) when L is 1. Considering that more network layers require more computational cost, we only construct one attention layer in the Feature Projector.
Number of Learnable Reference Representations N,. The results are shown in Tab 12 It can be found that with N, set to 2048, the PRO result under PatchCore is better than others, and the results under PaDiM and FeatureNorm representations is set to 2048 by default.
also good. Thus, the number of learnable reference
Number of Reference Samples during Training K. The results are shown in Tab 13 Compared to a fixed number of reference samples, combining multiple reference sample numbers and randomly selecting (Rand (1, 4, 8)) each time can achieve significant performance improvement. To vary the
27

---

Table 11: Ablation studies about the number of attention layers in the Feature Projector.
Method
PaDiM PatchCore FeatureNorm
95.5/88.1 94.6/87.5
95.7/88.2
95.7/88.4
94.9/86.4
95.0/86.7
94.0/88.2
94.0/88.1
93.9/88.2
Table 12: Ablation studies about the number learnable reference representations the Feature
Projector.
N
256
512
1024
1536
1792
2048
2560
Method
PaDiM
95,4/88. 1
95,6/88.3
95.5/88.
95.6/88.3
95.6/88.1
95.5/88.0
95.5/87.9
PatchCore
94.8/87.3
94.8/86.7
94.6/87.5
94.5/86.5
94.6/87.4
94.7/88.0
94.6/87.3
FoatureNorm
94.0/88.
94.2/88.3
94.0/88.2
93.8/88.0
93.6/87.9
93.8/88.1
93.7/88.1
number of reference samples during training, we think that it is beneficial for increasing residual feature diversity.
Table 13: Ablation studies about the number of reference samples used during training. Rand(I, 4, 8) means that we randomly select 1, 4, or 8 reference samples for each input sample.
K
1
2
8
Rand(1, 4, 8)
Method
PaDiM
95.5/88.0
95.3/88.0
95.3/88.1
95.2/88.7
95.5/88.8
PatchCore
94.7/88.0
94.4/87.5
94.6/87.7
94.3/87.5
94.7/87.6
FermureNorm
93.8/88.1
94.3/89.0
94.5/89.3
94.2/89.2
94.5/89.3
Qualitative Results
In this section, we show some qualitative results. To avoid redundant qualitative figures, we select to generate anomaly score maps based on PatchCore it is representative enough. PatchCore is based on feature comparison without learnable parameters, and thus can intuitively reflect the quality of pretrained features. In addition, we utilize CL.IP-I as the backbone network. The qualitative results are shown Fig 4 It can be seen that by employing our pretrained features, the PatchCore method generate better anomaly localization maps compared to using the original imageNettpretraine features. "PatchCore w/o pretrained" may generate many false positives in normal regions (e.g., line right part of lines 4 and 5). By comparison, "PatchCore w/ pretrained" can effectively avoid false positives in normal regions and locate anomalies more accurately (e.g., left part lines 3, 4, and 5).
In Fig we show more feature t-SNE visualization results.
28

---

Ground Truth
we pretrained
w/ pretrained
Ground Truth
we pretrained
wi pretrained
Mr/ecAD
MPDD
Figure 4: Qualitative results. The anomaly score maps are generated by PatchCore with CLIP-L as the backbone network. "w/o pretrained" and "w/ pretrained" refer to without and with our pretrained features.
Wio Pratrained
W/ Pretrained
Wio Pretrained
Wi Protr ained
(a) Candle
(b) Chewinggum
W/o Pretrained
W/ Pretrained
Wio Pretrained
Wi Protr reetrained
(c) Pcb1
(d) Fryum
Figure 5: Feature t-SNE visualization. For (a), (b), (c), and (d), the features are from the "candle". "chewinggum"" "pebl", and "fryum" classes from the VisA dataset.
29


Reviews:
[{"review_id": "8HNUaz5HBU", "reviewer": "Reviewer_yUBc", "summary": "This paper introduces ADPretrain, an anomaly-aware pretraining framework designed to generate better feature representations for unsupervised anomaly detection. The authors propose a pseudo-anomaly discrimination task along with multi-scale feature fusion to make the pretraining more aligned with downstream AD tasks.", "questions": "Could the authors compare ADPretrain against other pretrained representations such as MAE, DINO, or industrial-specific SSL models?\n\nWhat is the impact of pretraining dataset choice on generalization? Is ADPretrain truly domain-agnostic?\n\nGiven the add-on nature of ADPretrain, is it possible to integrate it into fully end-to-end AD pipelines (e.g., UniAD), rather than only use it for feature extraction?\n\nCould the authors elaborate on whether their method helps in resource-constrained scenarios (e.g., faster inference, fewer training images)?", "limitations": "The authors have clearly acknowledged the main limitations of their work, including the fixed backbone design and the restricted applicability to embedding-based AD methods. These reflections are fair and sufficient given the scope of the paper", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe key strengths of this paper is that it tackles a limitation in the current AD landscape.\nThe results are good. Across a wide variety of datasets and multiple backbones, the method improves performance. \n\nWeaknesses\n\n1.\tThe paper lacks direct comparisons against alternative pretrained backbones beyond ImageNet (e.g., MAE, DINOv2, self-distillation), which makes it difficult to contextualize the performance gain solely to the proposed pretraining strategy.\n2.\tThe method essentially acts as a drop-in feature encoder, and while the performance improvements are notable, the core contribution is mostly centered around representation learning rather than architectural novelty.\n3.\tThe paper would benefit from further analysis of computational cost, especially regarding whether ADPretrain can reduce downstream training/inference cost or improve sample efficiency.", "was_revised": true, "final_justification": "The authors clarified the method’s novelty and added useful experiments supporting its effectiveness and generalization. While I am not an expert in this specific subfield, the response seems reasonable and improves my overall impression of the work. I am therefore raising my score from 3 to 4.", "rebuttal": "**Dear Reviewer yUBc, due to time limitation and to respond to multiple reviewers, we currently don’t have enough time to provide experiment results as comprehensive as in the paper. In the revision, we will supplement the results on other datasets**.\n\n**[To W1]** Thanks for your professional review. We use the phrase ''ImageNet-pretrained'' not to mean that the backbones used in experiments are all from ImageNet pretraining, but to refer to visual feature networks pretrained on all sorts of large-scale datasets (not just ImageNet). Because ImageNet is the most well-known pretraining dataset, we thus still use ''ImageNet-pretrained''. We are sorry that this caused misunderstanding and will modify the phrase in the revision. The CLIP and ImageBind models (please see Table 1) used in experiments are based on self-supervised learning (i.e., contrastive language-image pretraining), and are not pretrained on ImageNet. Our experimental comparison way is to separately use the original features and our pretrained AD representations in an AD model (in Table 1, we conducted experiments on 5 AD models), train the AD model, obtain the results, and then compare the corresponding results based on the two features. We keep all other aspects of the model consistent, so the differences in results only stem from the different features. Therefore, the comparison way can directly reflect that our pretrained features are better than the original features when used in AD models. \n\nBased on your suggestion, we further conducted a set of experiments based on DINOv2-large, and the results are as follows. \n\n| Model | Datasets | PaDiM | PaDiM$^†$ | PatchCore | PatchCore$^†$ | CFLOW | CFLOW$^†$ | GLASS | GLASS$^†$ | UniAD | UniAD$^†$|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|DINOv2-large | MVTecAD | 98.7/91.0 | 98.6$\\textcolor{red}{-0.1}$/92.4$\\textcolor{green}{+1.4}$ | 97.8/84.9 | 98.0$\\textcolor{green}{+0.2}$/84.6$\\textcolor{red}{-0.3}$ | 98.8/92.7 | 98.9$\\textcolor{green}{+0.1}$/93.2$\\textcolor{green}{+0.5}$ | 98.4/95.3 | 99.1$\\textcolor{green}{+0.7}$/96.2$\\textcolor{green}{+0.9}$ | 96.6/89.3 | 97.1$\\textcolor{green}{+0.5}$/89.6$\\textcolor{green}{+0.3}$|\n| | VisA  | 92.6/85.6 | 95.1$\\textcolor{green}{+2.5}$/86.7$\\textcolor{green}{+1.1}$ | 84.2/71.4 | 85.9$\\textcolor{green}{+1.7}$/75.8$\\textcolor{green}{+4.4}$ | 96.2/90.0 | 96.9$\\textcolor{green}{+0.7}$/90.6$\\textcolor{green}{+0.6}$ | 93.3/90.4 | 94.0$\\textcolor{green}{+0.7}$/91.8$\\textcolor{green}{+1.4}$ | 87.6/85.8 | 90.0$\\textcolor{green}{+2.4}$/86.9$\\textcolor{green}{+1.1}$|\n\nThe results show that we can also achieve performance improvement based on DINOv2-large. Because there are a large number of experiments under one backbone, we currently don't have enough time to run all the backbones you mentioned. In the revision, we will complete the results and validate our pretraining method on other backbones you mentioned. In addition, please see our response to Weakness 3 of Reviewer rLun. We also provide the experiment results based on EfficientNet-b6.\n\n**[To W2]** We think that in the current development stage of anomaly detection, fundamental representation is more important than architectural novelty. In previous stages, many papers have proposed various architectural novelties, but overlooked that the essence of anomaly detection still can be attributed to the representation quality of features. In fact, architectural novelty is also aimed at better learning how to distinguish between normal and abnormal. If the normal and abnormal features are already highly discriminative (e.g., linearly separable), anomaly detection tasks can be easily accomplished without the need for designing increasingly complex architectural novelties. We also observe that most of the architectural novelties in AD papers are specific to the proposed method and are often hard to be still effective in other methods. However, good representations will be effective in various AD methods.\n\nThe current mainstream AD models mostly use pretrained networks to extract features. However, regardless of supervised or self-supervised pretraining on natural images, the pretraining process does not match the goal of anomaly detection. From the perspective of the development prospects of anomaly detection, we think that it's necessary for AD tasks to have specialized pretrained features instead of continuously using basic pretrained features. We think that pretrained representations specifically learned for anomaly detection are of great significance, and our work is only an early exploration. We hope to inspire more future works to focus on anomaly representation pretraining.\n\n**[To W3]** When our pretrained features are used in downstream AD models, feature extraction and specific AD modeling are still required. Thus, there is no reduction in inference cost. We further conducted an experiment (only 10\\% of the normal samples from each dataset are used for training), and obtained the following results (based on ImageBind). \n\n| Datasets | PaDiM | PaDiM$^†$ | PatchCore | PatchCore$^†$ |\n|-|-|-|-|-|\n| MVTecAD | 81.4/88.6 | 96.8$\\textcolor{green}{+15.4}$/90.3$\\textcolor{green}{+1.7}$ | 96.5/86.2 | 98.2$\\textcolor{green}{+1.7}$/87.7$\\textcolor{green}{+1.5}$ |\n| VisA | 82.8/79.3 | 93.0$\\textcolor{green}{+10.2}$/83.3$\\textcolor{green}{+4.0}$ | 88.9/78.7 | 93.9$\\textcolor{green}{+5.0}$/85.7$\\textcolor{green}{+7.0}$|\n\nThe results based on full training data are as follows.\n\n| Datasets | PaDiM | PaDiM$^†$ | PatchCore | PatchCore$^†$ |\n|-|-|-|-|-|\n| MVTecAD | 98.3/93.0 | 98.9$\\textcolor{green}{+0.6}$/93.2$\\textcolor{green}{+0.2}$ | 98.4/89.8 | 98.8$\\textcolor{green}{+0.4}$/89.8$\\textcolor{gray}{+0.0}$ |\n| VisA | 92.6/86.3 | 95.6$\\textcolor{green}{+3.0}$/88.6$\\textcolor{green}{+2.3}$ | 91.6/81.3 | 94.7$\\textcolor{green}{+3.1}$/88.2$\\textcolor{green}{+6.9}$ |\n\nThe results show that with less training data, our pretrained features can bring more significant performance improvement, indicating that our pretrained features are beneficial for the sample efficiency (during training) of downstream AD methods.\n\n**[To Q1]** Please see our response to Weakness 1.\n\n**[To Q2]** As pretraining is usually based on a large-scale dataset, and currently most AD datasets only have a few thousand samples, we thus choose RealIAD for pretraining due to its data scale. To answer your question, we further performed pretraining on the MVTecAD dataset. The obtained results are as follows (based on ImageBind).\n\n| Datasets | PaDiM(RealIAD) | PaDiM(MVTecAD) | PatchCore(RealIAD) | PatchCore(MVTecAD) |\n|-|-|-|-|-|\n| MVTecAD | 98.9/93.2 | 97.7/92.5 | 98.8/89.8 | 98.8/88.0 |\n| VisA | 95.6/88.6 | 93.0/85.6 | 94.7/88.2 | 93.4/86.0 |\n| BTAD | 95.9/76.8 | 95.6/76.6 | 95.6/67.3 | 94.7/66.5 |\n| MVTec3D | 84.4/92.0 | 82.8/91.2 | 82.6/87.0 | 80.6/87.3 |\n| MPDD | 94.4/95.1 | 92.9/93.5 | 94.8/94.2 | 93.1/93.4 |\n\nIt can be seen that with pretraining on MVTecAD, the results are overall lower than the results obtained based on RealIAD pretraining. Thus, we think that the scale of the dataset will have an impact, a larger scale is more advantageous for pretraining. This also reflects that the AD field still needs larger and higher-quality datasets in the future to better support anomaly representation pretraining. \n\nWe are not quite sure about your definition of domain-agnostic. Generally, domain-agnostic means that a trained model can also perform well in other domains or datasets. From this perspective, our pretrained features can achieve better results than the original features when used on 5 downstream datasets and 5 AD methods. This indicates that our pretrained features are domain-agnostic (at least better than the original features) and method-agnostic. If there is any deviation between our understanding of domain-agnostic and yours, we hope to further discuss with you in the subsequent discussion phase.\n\n**[To Q3]** Do you mean to combine our method with UniAD and train it on the RealIAD dataset? We think that residual features and the Feature Projector could be integrated into UniAD. These two contrastive losses could not be applied to UniAD because they are designed for representation learning, while the learning objective in UniAD is reconstruction. But we think that even if UniAD can perform well after training on RealIAD, it is only a specific AD model and cannot empower other AD models, while the pretrained fundamental representations are more valuable and can be used to enhance downstream AD models.\n\n**[To Q4]** Please see our response to Weakness 3.\n\n**We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you**."}, {"review_id": "a7wTWulr7g", "reviewer": "Reviewer_rLun", "summary": "This paper addresses the intriguing and under-explored topic of anomaly representation pretraining, a critical yet scarcely researched area in unsupervised anomaly detection. Existing state-of-the-art AD methods heavily rely on ImageNet-pretrained features, which suffer from mismatched objectives (e.g., classification vs. anomaly discrimination) and distribution shifts between natural and industrial images. The authors propose ADPretrain, a novel framework that leverages the large-scale RealIAD dataset to learn robust pretrained representations by integrating residual features and designing angle- and norm-oriented contrastive losses. This approach aims to maximize discriminative differences between normal and abnormal features while mitigating distribution shifts.  While the study acknowledges limitations such as reliance on embedding-based AD methods and fixed backbone networks, its focus on task-specific pretraining fills a significant research gap. By pioneering anomaly-focused pretraining, the work offers valuable insights and paves the way for future advancements, making it a promising contribution despite its room for improvement.", "questions": "- Compare ADPretrain with fine-tuning methods (e.g., FYD, PANDA) on the same datasets to quantify the benefit of task-specific pretraining.  \n- Introduce synthetic noise or corrupted samples in experiments to assess robustness.  \n- Validate the framework on emerging backbones (EfficientNet, DINOv2) to confirm its applicability across modern architectures.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### **Strengths**  \n- The paper addresses the limitations of using ImageNet-pretrained features for anomaly detection by proposing a task-specific pretraining framework, with well-designed residual features and contrastive losses tailored for AD.\n\n- The related work section provides a thorough review of existing methods and clearly positions the novelty of the proposed approach.\n\n\n### **Weaknesses**  \n-  While the paper emphasizes that ADPretrain outperforms ImageNet-pretrained features, it does not explicitly compare its framework with state-of-the-art fine-tuning methods like FYD or mean-shifted contrastive loss. Such a comparison is crucial to validate whether pretraining truly offers advantages over adaptive fine-tuning strategies in addressing distribution shifts .  \n\n-  The experiments do not test the framework under more settings, such as noisy conditions where normal samples might be corrupted. For example, methods like SoftPatch [1] have shown robustness to noise, but ADPretrain’s reliance on residual features—derived from clean normal references—may struggle if training data contains noise. This gap leaves uncertainty about its performance in real-world scenarios with imperfect data.  \n\n-  It does not evaluate modern architectures like EfficientNet or DINOv2, which have demonstrated superior representation capabilities in self-supervised learning. Additionally, while the paper mentions that residual features are class-generalizable, it does not explicitly validate their effectiveness on cutting-edge backbones (e.g., UniFormaly’s findings on ViT vs. CNN [2]). Expanding to more architectures would strengthen the generalizability claim.  \n\n[1] Jiang X, Liu J, Wang J, et al. Softpatch: Unsupervised anomaly detection with noisy data[J]. Advances in Neural Information Processing Systems, 2022, 35: 15433-15445.\n\n[2] Lee Y, Lim H, Jang S, et al. Uniformaly: Towards task-agnostic unified framework for visual anomaly detection[J]. Pattern Recognition, 2025: 111820.", "was_revised": true, "final_justification": "After several rounds of discussion, the author has addressed most of my questions, so I have decided to increase the score somewhat. I have a positive attitude toward the paper, but since the overall analysis is still not particularly impressive, my overall rating has not increased.", "rebuttal": "**Dear Reviewer rLun, due to time limitation and to respond to multiple reviewers, we currently don’t have enough time to provide experiment results as comprehensive as in the paper. In the revision, we will supplement the results on other datasets**.\n\n**[To W1]** As the FYD paper does not provide open-source code, we can only compare with the results provided in the paper. In FYD, anomaly detection is achieved based on PaDiM. The paper provides the results (with WideResNet50) on MVTecAD, which are 97.7/98.2 (image-level/pixel-level AUROCs). Based on WideResNet50 and PaDiM, the results of our method are 98.1/98.5. In PANDA and mean-shift contrastive loss (MSC), feature adaptation is based on global features, so only image-level anomaly detection is performed. The two papers provide image-level AUROC on MVTecAD, with values of 86.5 and 87.2. For a more reasonable comparison, we utilized the open-source code of MSC and trained an adapted network (based on WideResNet50) on MVTecAD. Then, we utilized PaDiM for AD modeling, and the obtained results are 96.8/97.0. Thus, compared with PANDA, MSC, and FYD, our method has advantages in providing better features for AD models. As discussed in Related Work, adaptive feature fine-tuning requires specialized fine-tuning on each dataset. By comparison, a more important advantage of anomaly representation pretraining is that pretrained features can be used on multiple downstream AD datasets without the need for fine-tuning on each dataset.\n\n**[To W2]** Thanks for your suggestion. We further conducted experiments with noisy data following the settings in SoftPatch. Specifically, we added abnormal data from the test set to the training set with a noise ratio of 0.1. We adopted the ''overlap'' setting from the SoftPatch paper, which means the injected abnormal images are also included in the test set. The results are as follows (based on ImageBind, image-level AUROC/PRO).\n\n|Datasets|PaDiM|PaDiM$^†$|PatchCore|PatchCore$^†$|\n|-|-|-|-|-|\n|MVTecAD|86.6/82.7|88.2$\\textcolor{green}{+1.6}$/84.6$\\textcolor{green}{+1.9}$|87.7/80.6|89.2$\\textcolor{green}{+1.5}$/81.4$\\textcolor{green}{+0.8}$|\n|VisA|82.8/75.6|87.4$\\textcolor{green}{+4.6}$/79.9$\\textcolor{green}{+4.3}$|83.4/71.5|86.9$\\textcolor{green}{+3.5}$/78.7$\\textcolor{green}{+7.2}$|\n\nThe results without noisy data are as follows.\n\n|Datasets|PaDiM|PaDiM$^†$|PatchCore|PatchCore$^†$|\n|-|-|-|-|-|\n|MVTecAD|98.3/93.0|98.9$\\textcolor{green}{+0.6}$/93.2$\\textcolor{green}{+0.2}$|98.4/89.8|98.8$\\textcolor{green}{+0.4}$/89.8$\\textcolor{gray}{+0.0}$|\n|VisA|92.6/86.3|95.6$\\textcolor{green}{+3.0}$/88.6$\\textcolor{green}{+2.3}$|91.6/81.3|94.7$\\textcolor{green}{+3.1}$/88.2$\\textcolor{green}{+6.9}$|\n\nIt can be seen that the performance of the AD models significantly decreases when there is noisy data. Although the performance of the AD models with our pretrained features also decreases (this is normal, our paper is not intended to tackle the contamination AD task). However, our pretrained features can bring greater performance improvement under noisy data. This indicates that our pretrained features are more robust to noise than the original ImageNet-pretrained features. Moreover, our method is compatible with contamination AD methods, such as SoftPatch. When the training data has noise, we can first utilize the filtering method in SoftPatch to remove the most likely abnormal data, and then train based on the remaining data.\n\nFinally, we note that for the above results, we don't add abnormal data to the normal reference samples. This is because when generating residual features, we only used few-shot and fixed normal samples as reference (i.e., 8 samples). We think that in real-world scenarios, even if training data has noise, it's easy and almost costless to manually collect few-shot normal samples as reference. Thus, we think that it seems not necessary to forcefully introduce abnormal data to the few-shot normal reference samples. In the revision, we will add the above results and discussion and cite the paper you mentioned.\n\n**[To W3]** Thanks for your suggestion. Based on your suggestion, we further conducted a set of experiments based on EfficientNet-b6 and DINOv2-large, and the results are as follows (image-level AUROC/PRO).\n\n|Model|Datasets|PaDiM|PaDiM$^†$|PatchCore|PatchCore$^†$|CFLOW|CFLOW$^†$|GLASS|GLASS$^†$|UniAD|UniAD$^†$|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|EfficientNet-b6|MVTecAD|95.3/87.8|96.8$\\textcolor{green}{+1.5}$/88.5$\\textcolor{green}{+0.7}$|98.6/92.0|98.9$\\textcolor{green}{+0.3}$/92.5$\\textcolor{green}{+0.5}$|97.9/91.6|98.5$\\textcolor{green}{+0.6}$/92.4$\\textcolor{green}{+0.8}$|98.7/93.6|99.0$\\textcolor{green}{+0.3}$/94.7$\\textcolor{green}{+1.1}$|96.9/89.5|97.7$\\textcolor{green}{+0.8}$/90.7$\\textcolor{green}{+1.2}$|\n| |VisA|86.4/76.7|90.2$\\textcolor{green}{+3.8}$/84.3$\\textcolor{green}{+7.6}$|93.5/87.7|94.0$\\textcolor{green}{+0.5}$/88.3$\\textcolor{green}{+0.6}$|93.3/86.1|93.7$\\textcolor{green}{+0.4}$/86.8$\\textcolor{green}{+0.7}$|94.2/88.6|94.7$\\textcolor{green}{+0.5}$/89.6$\\textcolor{green}{+1.0}$|92.7/85.9|94.3$\\textcolor{green}{+1.6}$/87.2$\\textcolor{green}{+1.3}$|\n|DINOv2-large|MVTecAD|98.7/91.0|98.6$\\textcolor{red}{-0.1}$/92.4$\\textcolor{green}{+1.4}$|97.8/84.9|98.0$\\textcolor{green}{+0.2}$/84.6$\\textcolor{red}{-0.3}$|98.8/92.7|98.9$\\textcolor{green}{+0.1}$/93.2$\\textcolor{green}{+0.5}$|98.4/95.3|99.1$\\textcolor{green}{+0.7}$/96.2$\\textcolor{green}{+0.9}$|96.6/89.3|97.1$\\textcolor{green}{+0.5}$/89.6$\\textcolor{green}{+0.3}$|\n| |VisA|92.6/85.6|95.1$\\textcolor{green}{+2.5}$/86.7$\\textcolor{green}{+1.1}$|84.2/71.4|85.9$\\textcolor{green}{+1.7}$/75.8$\\textcolor{green}{+4.4}$|96.2/90.0|96.9$\\textcolor{green}{+0.7}$/90.6$\\textcolor{green}{+0.6}$|93.3/90.4|94.0$\\textcolor{green}{+0.7}$/91.8$\\textcolor{green}{+1.4}$|87.6/85.8|90.0$\\textcolor{green}{+2.4}$/86.9$\\textcolor{green}{+1.1}$|\n\nIt can be seen that we can also achieve performance improvement on EfficientNet-b6 and DINOv2-large. In the revision, we will supplement the results on other datasets and cite the paper you mentioned.\n\n**[To Q1]** Please see our response to Weakness 1.\n\n**[To Q2]** Please see our response to Weakness 2.\n\n**[To Q3]** Please see our response to Weakness 3.\n\n**We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you**.\n***\n***\n***\n***\n***\n***\n***\n***\n***\n***\n**Note: [To Reviewer rLun], you don't need to see the following content**. Due to space limitation, we borrow some space to answer some questions of Reviewer oLCH. We sincerely apologize for any inconvenience caused to you.\n\n**[To W4]** (2) As stated in Abstract, we promise that we will open-source the code, it's not to worry. When submitting the paper, we didn't organize all code well (with sufficient instructions), so we were unable to directly include it in the supplementary materials.\n\n**[To Q1]** In the Norm-Oriented Contrastive Loss, the basic idea follows one-class-classification (OCC) learning. In OCC learning, normal features are usually optimized to be located inside the unit hypersphere (i.e., when $||x_i|^2_2 = 1$, it locates on the unit hypersphere). In the pseudo-Huber distance $\\sqrt{||x_i|^2_2+1}-1$, when $||x_i|^2_2 = 1$,  $\\sqrt{||x_i|^2_2+1}-1 \\approx 0.4$. Setting $r$ to 0.4 is equivalent to the unit hypersphere based on Euclidean distance.\n\n**[To Q2]** WideResNet50 outputs two layers of features, with dimensions of 512 and 1024. The features generated by CLIP-base/CLIP-large/ImageBind are all 768/1024/1280 dimensions. The feature dimension has no direct relation to detection performance. Detection performance mainly depends on the feature network. It's not that with a larger feature dimension, the detection performance must be better.\n\n**[To Q3]** In line 274 to 275, we mentioned that \"we further propose a simple AD baseline (denoted by FeatureNorm), where we directly utilize feature norms as anomaly scores\". For few-shot AD, our FeatureNorm is really simple. Specifically, for an input sample, we extract pretrained residual features from our pretrained model. Then, for a pretrained residual feature $x_i$, the L2 norm $||x_i||_2$ of $x_i$ is used as the anomaly score. In the revision, we will add this more specific description to the paper.\n\n**[To L1]** We further conducted experiments based on OCSVM and KNN, and the results are as follows (based on ImageBind).\n\n|Datasets|OCVSM|OCVSM$^†$|KNN|KNN$^†$|\n|-|-|-|-|-|\n|MVTecAD|84.8/77.9|90.4$\\textcolor{green}{+5.6}$/82.8$\\textcolor{green}{+4.9}$|92.6/84.7|97.8$\\textcolor{green}{+5.2}$/88.4$\\textcolor{green}{+3.7}$|\n|VisA|75.3/72.2|81.4$\\textcolor{green}{+6.1}$/77.3$\\textcolor{green}{+5.1}$|80.4/75.6|87.5$\\textcolor{green}{+7.1}$/80.2$\\textcolor{green}{+4.6}$|\n\nIt can be seen that our pretrained features can achieve significant performance improvement, which further indicates that our pretrained features have better discriminability between normal and abnormal. Due to time limitation, we only conducted experiments on two datasets. In the revision, we will supplement experiments on other datasets and try other algorithms.\n\n**[To L2]** In Tab.2(b), we have conducted ablation studies about MLP (the MLP Projector in Tab.2(b)). Does the ''readout layer'' refer to the network layers that output features? We further conducted experiments, and the results are as follows.\n\n|Architecture|PaDiM|PatchCore|FeatureNorm|\n|-|-|-|-|\n|Output Layer Learnable|94.0/87.9|93.8/86.5|93.2/85.7|\n|Learnable Key/Value Attention (ours)|95.5/88.8|94.7/87.6|94.5/89.3|\n\nCompared to training the whole network, only training the output layers can significantly improve the results. This may be because fixing most parameters can effectively preserve the basic visual representation capabilities possessed in the pretrained backbone, which are still valuable and helpful. We greatly appreciate your suggestion. In the revision, we will include these ablation studies you mentioned."}, {"review_id": "dsYKSqCKZ2", "reviewer": "Reviewer_SPbR", "summary": "This paper proposes a new contrastive learning strategy and feature adaptation network focusing on industrial anomaly detection scenarios. The ability of residual features is fully exploited, and Learnable Key/Value Attention is proposed for feature adaptation.\nThen, Angle-Oriented Contrastive Loss and Norm-Oriented Contrastive Loss are designed to expand the distinction between normal and abnormal features. The sufficient experimental performance improvement on five public datasets proves the advantages of this method. And it can also enhance the adaptability of different types of feature extractors to industrial scenarios.", "questions": "See weaknesses.\n\nIf the authors could solve my questions, especially weaknesses 2, 4, 7 and 9, I will consider further improving my rating.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The authors propose two novel contrastive learning losses, Angle-Oriented Contrastive Loss and Norm-Oriented Contrastive Loss, and the theoretical analysis of the two losses is very thorough.  The ablation studies and t-SNE prove that these two losses could increase the difference between normal and abnormal features, thereby improving the performance of the current anomaly detection model.\n2. Although residual features are not proposed for the first time in this paper, the analysis and application of residual features in this paper are thorough.  Moreover, the paper designs an effective Learnable Key/Value Attention to transform this feature for subsequent representation learning.\n3. The authors conduct sufficient experiments, proving that this training strategy could bring improvements for different AD methods and feature extractors of different architectures.\n4. The issue focused on by this paper has great significance to the industrial field. This work could fill the gap in open-source work on feature adaptation in the industrial AD tasks.\n\nWeaknesses:\n1. The ablation studies in Table 2 indicate that the non-fixed feature extractor leads to significant performance degradation. This phenomenon means that the weights obtained through pre-training in natural scenarios are very important. This work only adapts the original pre-trained features to industrial scenarios, which is difficult to be defined as \"pretrain\".\n2. line73: \"this is the first study dedicated to anomaly representation pertaining.\" However, the paper **spot-the-difference**[1] proposes a special data augmentation strategy on natural images and uses contrastive learning for pre-training for subsequent industrial anomaly detection. Although the method proposed in this paper is quite different from spot-the-difference, it is beneficial to discuss it in the related works.\n3. Why is only the GLASS-h variant used for evaluation? Is it because the Manifold Hypothesis in GLASS-m contradicts the Hypersphere-based loss in this paper?\n4. The products in the Real-IAD dataset are all \"object classes\". Therefore, providing some qualitative and quantitative results on the \"texture classes\" (such as carpet, tile and grid on the MVTec AD dataset) will be beneficial for better evaluating the generalization performance of features.\n5. line 188: \"we can downsample the mask to get feature labels.\" What is the specific downsampling operation (max pooling or average pooling or others) ? Different downsampling strategies could lead to some patches being assigned different labels.\n6. Are the output features of the Feature Projector used to calculate both the Angle-Oriented Contrastive Loss and the Regul-oriented Contrastive Loss?  The symbols in this paper are not consistent and clear.\n7. In the Learnable Key/Value Attention, the \"addition operation\" are replaced to \"subtraction operation\". This improvement requires more explanations and motivations. It seems like performing a secondary residual calculation on the original residual features, but the original residual features have already highlighted the abnormal regions.\n8. In the inference phrase, some normal samples need to be provided for calculating residual features, which might limit the wider application of this framework, such as zero-shot and contaminated AD tasks without labeled normal data.\n9. In few-shot (especially 2-shot) settings, the influence of reference samples on residual features needs to be discussed. For example, for the screw class of the MVTec AD dataset, samples with different rotation directions are adopted as reference samples. Or, reference samples could be selected through different random seeds for sensitivity analysis.\n\nSome extended problems:\n1. Whether the features extracted through the framework in this paper will eliminate the original pre-trained model's ability to distinguish different types of anomalies.\n2. Whether it can be applied to supervised anomaly detection tasks by adding a classification head or a segmentation head, such as the experiments in the paper \"Spot-the-difference\". Although abnormal images are difficult to collect, in real industrial scenarios, supervised anomaly detection models still have very wide applications at present.\nThe responses to these extended questions will not lead to my negative evaluation of this paper. This paper has provided enough contribution to the AD community.\n\n[1] Zou Y, Jeong J, Pemula L, et al.  Spot-the-difference self-supervised pre-training for anomaly detection and segmentation[C]//European Conference on Computer Vision.  Cham: Springer Nature Switzerland, 2022: 392-408.", "was_revised": true, "final_justification": "After carefully evaluating the authors' responses, I am pleased to confirm that all my concerns have been addressed. While the rebuttals for W3, W5, W8, and W9 do reveal certain limitations in the current work, I recognize this study as a valuable exploration along the path of developing \"pretrained industrial anomaly detection models\". Considering both the technical contributions and the authors' conscientious rebuttal, I have decided to elevate my rating from Borderline Accept (Rating 4) to Accept (Rating 5).", "rebuttal": "**[To W1]** We think that the reasons for the performance degradation in the non-fixed backbone may be: (1) the basic visual representation capabilities possessed in the pretrained backbone are still valuable and important, and training the whole feature network on AD datasets may cause impairment to the basic visual representation capabilities. (2) The current scale and quality of AD datasets are not enough to support the whole network training of an AD backbone. The use of the word ''pretrain'' has two reasons: (1) We are indeed following the basic paradigm of pretraining: learning on a large-scale dataset, and the learned features can be transferred to downstream datasets and models. (2) More importantly, we want to emphasize that in the current development stage of anomaly detection, pretrained representations specifically learned for anomaly detection are of great significance.  If it is revised to “pretrained features finetuning”, the significance of this research direction cannot be well reflected.\n\n**[To W2]** In the paper spot-the-difference (SPD), the authors propose a data augmentation strategy called SmoothBlend to create slight local differences (or called pseudo anomalies). Then, the authors create pseudo anomalies on natural images from ImageNet to use as negative samples in contrastive learning. The model is pretrained on ImageNet, while our model is pretrained on a real AD dataset, RealIAD. (1) Compared to pseudo anomalies, the negative samples used in our method are more realistic and more in line with the abnormal characteristics in real-world scenarios. (2) In SPD, similar to classical SSL methods (*SimCLR*, *MoCo*), contrastive loss is constructed based on global features, while we construct contrastive loss based on more fine-grained local features. Although SPD mentioned that it can promote local sensitivity compared to *SimCLR*, *MoCo*, etc., it still may lead to the loss of information about local pseudo anomalies in the global features, as the local augmentation is slight. For fine-grained task like anomaly detection, contrastive learning based on fine-grained local features should be more suitable. (3) SPD was evaluated only on the ResNet-50, PaDiM, and VisA and MVTecAD datasets, while we evaluate our method on more backbones, AD models, and AD datasets. \n\nAs the SPD paper was accompanied by the introduction of the VisA dataset, and subsequent works mainly cited this paper due to the VisA dataset, we inadvertently overlooked the method. We greatly appreciate your suggestion and will add a discussion with SPD in the revision.\n\n**[To W3]** Yes, at that time, we mainly felt that the hypersphere hypothesis in GLASS-h was more compatible with our norm-oriented contrastive loss. In the revision, we will also attempt to conduct experiments based on GLASS-m.\n\n**[To W4]** Thanks for your suggestion. In the following two tables, we report the results (image-level AUROC/PRO) on carpet, grid, leather, tile, and wood classes from MVTecAD. The first table is based on CLIP-large, and the second table is based on ImageBind.\n\n| Classes | PaDiM | PaDiM$^†$ | PatchCore | PatchCore$^†$ |\n| - | - | - | - | - |\n| Carpet | 93.9/96.2 | 100+6.1/96.6+0.4 | 98.5/90.7 | 100+1.5/91.7+1.0|\n| Grid | 96.7/91.1 | 99.2+2.5/92.5+1.4 | 91.1/75.7 | 98.6+7.5/79.5+3.8|\n| Leather | 98.4/97.2 | 100+1.6/97.8+0.6 | 100/96.0 | 100+0.0/95.5-0.5|\n| tile | 93.4/84.5 | 99.6+6.2/85.1+0.6 | 99.3/78.4 | 99.7+0.4/81.9+3.5|\n| wood | 98.2/88.7 | 99.5+1.3/91.0+2.3 | 96.9/76.7 | 99.7+2.8/79.8+3.1 |\n\n\n|Classes|PaDiM|PaDiM$^†$|PatchCore|PatchCore$^†$|\n|-|-|-|-|-|\n| Carpet | 100/96.4 | 100+0.0/97.2+0.8 | 100/94.0 | 100+0.0/93.8-0.2|\n|Grid|98.5/93.0|99.7+1.2/93.6+0.6 | 98.8/87.9 | 100+1.2/89.4+1.5|\n|Leather | 99.9/97.5 | 100+0.1/97.7+0.2 | 100/96.1 | 100+1.6/96.7+0.6|\n|tile | 98.8/85.6 | 99.1+0.3/85.1-0.5 | 98.2/83.0 | 98.4+0.2/82.6-0.4|\n|wood | 99.1/92.3 | 99.5+0.4/92.7+0.4 | 98.1/85.2 | 99.1+1.0/86.2+1.0|\n\nIt can be found that the results on texture classes can also be effectively improved. We are sorry that we cannot provide you with qualitative results directly, as the rebuttal text box does not support uploading images. Previously, NeurIPS supported submitting a one-page PDF file to upload images, but this function was canceled this year. In the revision, we will add the corresponding qualitative results.\n\n**[To W5]** For the feature map with a downsampling ratio of $s$, we divide the original mask into patches with $s\\times s$ size. Then, for each patch, we sum the number of 1s (1 represents abnormal in the ground-truth mask). If the number of 1s is larger than 32, we label this patch as 1. Compared to ''nearest'' downsampling, our downsampling method can preserve more patches labeled as 1. Because the number of abnormal patches is much smaller than that of normal patches, we found that the ''nearest'' downsampling can cause instability in the angle-oriented contrastive loss (*i.e.*, NaN).\n\n**[To W6]** Yes, the features yielded by the Feature Projector are used to calculate both the Angle-Oriented Contrastive Loss and the Norm-Oriented Contrastive Loss. \n\n**[To W7]** In line 333 to 340, we provided some explanations to our Learnable Key/Value Attention based on the results in Tab.2(b). In Tab.2(b), for Cross Attention, we convert normal reference features into residual features and then utilize them as Key and Value. Compared to Self Attention, Cross Attention can perform better. Thus, we think that only embodying normal patterns in Key and Value should be effective. Then, we further adopt learnable Key and Value. Compared to limited normal patterns in Cross Attention, the learnable reference should can adaptively learn to represent normal patterns and better cover key normal feature patterns in the residual feature distribution during pretraining. By subtraction, we aim to eliminate the normal representations in the residual feature distribution adaptively learned by the network, further increasing the discrepancy between normal and abnormal residual features.\n\n**[To W8]** The need for some normal samples is a limitation mainly from residual features. However, we think that collecting a small amount of uncontaminated normal samples is usually feasible and easy for practical use. From the essence of anomalies (defined relative to normal), we think few-shot is more reasonable than zero-shot because at least we need to know what is normal and then detect anomalies. About contamination AD, you can see our response to Weakness 2 of Reviewer rLun, we provide some results and a discussion.\n\n**[To W9]** We greatly appreciate your suggestion. Residual features are mainly relied on mutually eliminating class-related normal patterns in features through subtraction (you could see the ResAD paper for more explanations), for being class-generalizable. However, if normal images have significant variability, it may cause the few-shot reference samples to be not representative enough (lacking some class-related normal patterns). This may lead to certain class-related normal patterns in the input feature can't be effectively eliminated. Then, the discriminability of residual features under few-shot settings will be compromised. \n\nFor practical applications, this issue should be particularly focused and reasonably addressed. Of course, the simplest solution is to increase the number of reference samples. This is feasible, as in practical applications, the number of reference samples is usually not as strict as the 2-shot. However, in practical applications, we expect that the reference samples can fully represent their class, so it’s best to have sufficient differences between the reference samples. Thus, the sample selection strategy cannot be random. A feasible method is to first cluster all available normal samples into different clusters based on a clustering algorithm (e.g., KMeans). Then, based on the number of reference samples, we evenly distribute it to each cluster. When selecting from a cluster, we can prioritize selecting samples closer to the center. Another feasible strategy is to match spatially aligned samples as reference samples for each input sample. This will be helpful for the example you mentioned, as we can match samples with similar rotation directions to the object in the input sample as reference samples. We have provided a corresponding method in Appendix B.\n\nIn the revision, we will add the above discussion about the influence of reference samples on residual features and the sample selection strategy to the paper.\n\n**[To Q1]** Yes, because one characteristic of residual features is that the semantic discriminability in features will be eliminated. However, for anomaly detection, it is okay not to distinguish different types of anomalies, and most AD methods can only detect anomalies and cannot give anomaly types.\n\n\n**[To Q2]** We think that our pretrained features can be used for supervised anomaly detection tasks. In supervised AD tasks, the model is more likely to learn good decision boundaries between normal and abnormal based on our pretrained features by effectively utilizing the abnormal samples during training. Because our pretrained features have better discriminability for representing normal and abnormal compared to the original features. Due to time limitation, we currently don't have enough time to provide the corresponding experimental results. In the revision, we will attempt to apply our pretrained features to supervised AD methods.\n\n**We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you**."}, {"review_id": "QR0qPFqDsY", "reviewer": "Reviewer_oLCH", "summary": "This work proposes a feature extractor grounded in existing pretrained model (such as ResNet50, CLIP on ImageNet) for industrial anomaly detection (IAD). The proposed framework designs an angle- and norm-oriented contrastive loss to train the Feature Projector based on RealIAD, a large industrial anomaly detection dataset. The empirical results indicate that the features produced by the trained Feature Projector improve the detection performance of five embedding-based AD methods in comparison to using their original features.", "questions": "1. How to determine the hyperparameter radius r? Authors have mentioned that it was set to 0.4 in the experiments.\n\n2. How large is the feature dimension? Does different dimension have a significant impact on detection performance?\n\n3. It needs to be clarified how to construct a few-shot AD method by the feature norm. \n\nOther questions see Weaknesses", "limitations": "1. Building on the optimization objective and empirical results, the features produced by the proposed framework show the strong distinguishability. In such situations, it should be explored when directly applying the feature vectors to the classical AD algorithms, such as OCSVM, LOF, Isolation Forest, KNN etc. It is possible to obtain good performance due to the results of FeatureNorm. If the features indeed achieve better or comparable performance on the conventional AD algorithms than other features even the end-to-end deep methods, the proposed framework will produce much higher social impact due to the robustness and low computational cost of these conventional methods.\n\n2. Two simple Feature Projector designs should be explored, (1) concatenating a MLPs (learnable parameters) behind the pretrained model (fixed parameters) and then fine-tuning the MLPs; (2) fine-tuning only the readout layer of the pretrained model when freezing all other parameters", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed framework effectively utilizes the RealIAD and the existing pretrained feature extractors for industrial anomaly detection.\n\n2. The empirical results demonstrate the effectiveness of the proposed framework.  \n\n\n**Weaknesses:**\n\n1. Ambiguous and misleading writing and statements\n\n(1)\tThe “Unsupervised Anomaly Detection” is highlighted in Title of the manuscript. However, the proposed framework is a feature extractor, does not direct relationship with “Unsupervised learning” and the training process of the Feature Projector is based on ground truth of RealIAD (supervised learning). In fact, the proposed method is only designed to obtain the features and it doesn’t care about the specifics of the downstream AD task, whatever supervised or unsupervised.\n\n(2)\tThis work only focuses on industrial anomaly detection (IAD). Thus, the research field should be clarified rather than directly using “anomaly detection” in the beginning. For instance, the description in line 1-3 is incorrect when considering other data types (time-series, graph, tabular) or application scenarios.\n\n(3)\tThis work highlights the “Pretraining” in the many places of the manuscript, which makes me think a totally novel pretrained model for IAD in the beginning. However, the experimental settings show that the backbone (architecture) and even the parameters of the pretrained models are remained accurately. In other words, this work just fine-tunes the existing pretrained models on a vertical field dataset (RealIAD) for industrial anomaly detection via slotting a Feature Projector. I suggest that authors revise some statements.\n\n2. Unclear technique details\n\n(1)\tOn Angle-Oriented Contrastive Loss: \nThe training process of Feature Projector is supervised, that is it is easy to get positive-pairs and negative-pairs, why still using data augmentation to produce more positive-pairs? And, how to ensure the augmented samples closer to normal data than abnormal data? Especially, if the forming of anomalies is similar with the used data augmentation techniques, for example, food spoilage and random color jitters, it may cause the wrong optimization direction.\n\n(2)\tNorm-Oriented Contrastive Loss:\nWhy using pseudo-Huber distance? Any defects or problems if using a simple Euclidean? I understand the first term of the loss, but why adding a multiplicative term $e^{d_j}$?\n\n(3)\tLearnable Key/Value Attention and Feature Projector:\nThe current contents are hard to understand in Section 3.3. To improve the readability, it should describe the function of feature projector using standard formulars step by step.\nI am curious about the learnable reference. After training, whether it was be fixed for extracting features for new datasets? How to understand the learnable reference?\n\n(4)\tAuthors need to check and clarify the optimized object. In line140, which one is the residual representation, x or r? And all the loss functions use x as optimized object. \n\n3. The fairness of comparison\n\nIn Appendix D Hyperparameter Sensitivity, authors explore the effects of different hyperparameter settings for detection performance. There are several concerns as follows:\n\n(1)\tWhat and where is the test data?\n\n(2)\tIt is unreasonable that the hyperparameters selection depends on the performance on the used embedding-based AD methods. The proposed framework is a feature extractor and it should be method-agnostic, like a pretrained ResNet-50 on ImageNet rather than method-dependent, otherwise, the proposed framework is just tailored for certain AD methods.\n\n(3)\tWhy does the performance look like quite robust (<1% fluctuation) in almost all cases for the changing of the hyperparameters?\n\n(4)\tFor selection of $N_r$, I noticed the performances are quite close for $N_r=256$ and $N_r=2048$. It should extend the range from 1-256 to more suitable selection.\n\n4. Statistical significance and reproducibility of the results\n\n(1)\tThe empirical results do not report any average values and standard variance. \n\n(2)\tThe code is not provided.", "was_revised": true, "final_justification": "I acknowledge the authors’ thoughtful response. The rebuttal has addressed some of my concerns, especially the presented results regarding $N_r$ and the conventional AD methods.  Based on this consideration, I decide to upgrade my score.", "rebuttal": "**[To W1]** (1) The use of \"unsupervised anomaly detection\" mainly refers to downstream AD tasks, as unsupervised AD is the most basic paradigm in anomaly detection (AD), and our experiments are also based on unsupervised AD methods. This \"unsupervised\" doesn't mean that the training process is unsupervised. \n\n(2) Combined with (1), we think a good revision would be to use \"industrial anomaly detection\" in the title and the main text.\n\n(3) The use of \"pretraining\" has two reasons: 1) We are indeed following the basic paradigm of pretraining: learning on a large-scale dataset, and the learned features can be transferred to downstream datasets and models. However, fine-tuning generally refers to fine-tuning a model on a dataset, and then applying the model to the dataset. We think that pretraining should not limit that must train the whole network. 2) More importantly, we want to emphasize that in the current development stage of AD, pretrained representations specifically learned for AD are of great significance. If it's revised to \"pretrained features fine-tuning\", the significance of this research direction cannot be well reflected. Current mainstream AD models mostly use pretrained networks to extract features. However, regardless of supervised or self-supervised pretraining on natural images, the pretraining process does not match the goal of AD. From the perspective of the development prospects of AD, it's necessary for AD tasks to have specialized pretrained features instead of continuously using basic pretrained features. We hope to inspire more future works to focus on anomaly representation pretraining. \n\n**[To W2]** (1) **We respectfully argue that you may misunderstand our Angle-Oriented Contrastive Loss**. The data augmentation is to create a positive pair for each feature $x_i$. You may think that for a normal/abnormal feature, we use all other normal/abnormal features as positive pairs. But it's not like that (you can look the Eq.(2) again). For $x_i$, we only use $x_{i\\prime}$ ($x_{i\\prime}$ is the feature from the augmented image, its position in the augmented image is the same as the position of $x_i$ in the original image) as the positive pair. Other features with different labels from $x_i$ are used as negative pairs. This ensures that we only perform contrast between normal and abnormal. This is the same as in classical contrastive learning methods, where for a sample, only the augmented sample is treated as the positive pair, while other samples are treated as negative pairs.\n\nAnomalies are usually different from each other, so abnormal features cannot be mutually used as positive pairs to attract together. For normal, there are also differences among different normal patches, so normal features also should not be mutually used as positive pairs.  Thus, for $x_i$, we only use the feature $x_{i\\prime}$ as the positive pair. Another reason is that if we use all features with the same labels to $x_i$ as positive pairs, this will cause numerical issues in the loss (i.e., appearing NaN).\n\nIn Eq.(2), we only use $x_i$ as the anchor feature, without using $x_{i\\prime}$ as the anchor feature. That is to say, there will be no contrast between the augmented image and abnormal data in our loss. You can simply think about the fact that if $x_{i\\prime}$ is not used as the anchor feature, there will be no negative pairs formed between $x_{i\\prime}$ and abnormal features. Thus, there's no need to worry about the issue you mentioned.\n\n(2) In line 197, we mentioned that the previous work FCDD [23] stated that the pseudo-Huber distance is a more robust distance measure that interpolates from quadratic to linear penalization; we follow to use this distance. Mathematically, when the value $x$ decreases from large to small, the curve of the pseudo-Huber function will change from quadratic to linear, while the curve of the Euclidean function is always quadratic. When $x$ is small, linear function can provide better gradient information compared to quadratic function. The term $e^{d_i}$ is a weighting factor. In line 205 to 209, we provided a good explanation. When $d_i$ is large (i.e., features are far from the hypersphere), $e^{d_i}$ can provide more weighting. When features are closer to the origin (i.e., $d_i \\rightarrow -r$),  $e^{d_i}$ can provide less weighting. Thus, the loss in Eq.(3) can adaptively assign larger gradients to the features outside the hypersphere for better contraction.\n\n(3) In Fig.2, we provided a clear diagram of the Feature Projector, it's not hard to understand when combined with the diagram. The Feature Projector is actually cross-attention combined with MLP, but we use learnable reference representations as Key and Value. \n\nIn line 333 to 340, based on Tab.2(b), we provided a clear explanation. In Tab.2(b), for Cross Attention, we convert normal reference features into residual features and then utilize them as Key and Value. Compared to Self Attention, Cross Attention can perform better. Thus, we think that only embodying normal patterns in Key and Value should be effective. Then, we further adopt learnable Key and Value. Compared to limited normal patterns in Cross Attention, the learnable reference can adaptively learn to represent normal patterns and better cover key normal feature patterns. For downstream datasets, it is fixed to ensure the Feature Projector can properly generate good pretrained features. We attempted to set it as learnable, but the features yielded by the Feature Projector will appear pattern collapse issue.\n\n(4) For all losses, $x$ means residual features. In Sec.3.2 and 3.3, we use $x$ as it is commonly used to represent features. In the revision, we will add a note to avoid confusion.\n\n**[To W3]** (1) In line 311 to 313, we stated that all ablation studies were based on the VisA dataset. Thus, the test data comes from the VisA dataset.\n\n(2) **I respectfully argue that our method is not method-dependent, and you may misunderstand the meaning of embedding-based AD methods in our paper**. In the introduction, we explained that we use embedding-based AD methods to refer to AD methods that use pretrained feature networks. So, the embedding-based AD methods in our paper actually cover a wide range. Like the pretrained ResNet-50 on ImageNet, our pretrained features can be applied to any AD methods that use pretrained feature networks. Perhaps this term ''embedding-based'' caused the misunderstanding. In the revision, we will make better modifications for this.\n\nIn Tab.1, we utilized 5 AD methods, including PaDiM and PatchCore (feature comparison based), CFLOW (normalizing flow based), GLASS (anomaly synthesis based), and UniAD (reconstruction based), to comprehensively validate the effectiveness of our method. This also demonstrates that our method is not method-dependent. In ablation studies, we evaluate the results based on PaDiM and PatchCore. We also explained in line 269 to 270: these two methods are based on feature comparison without learnable parameters. Thus, the results of these two methods directly depend on the quality of the representation. The better the discriminability between normal and abnormal features, the better the results. Thus, these two methods can directly reflect the quality of pretrained features. For hyperparameter experiments, it's costly for us to evaluate the results on all 5 AD methods, so we think that using PaDiM and PatchCore is the most appropriate.\n\n(3) We think that our contrastive losses are not very sensitive to the three loss hyperparameters. As we found that different loss hyperparameters all can effectively reduce the loss value and converge to a close loss value. This indicates that the optimization effect of the model after loss convergence is similar, so its performance is also close. Compared to the loss hyperparameters, the effects of $N_r$ (see the following (4)) and $K$ are more obvious. In Table 10, the results of Rand(1, 4, 8) and 1 are 94.5/89.3 v.s. 93.8/88.1 under FeatureNorm.\n\n(4) Thanks for your suggestion. We further conducted experiments with $N_r$ set to 32, 64, and 128, and the results are as follows.\n\n|Method|32|64|128|256|2048|2560|3072|4096|\n|-|-|-|-|-|-|-|-|-|\n|PaDiM|92.9/86.4|93.7/87.3|95.2/87.9|95.4/88.1|95.5/88.0|95.5/87.9|95.2/87.6|94.4/87.1|\n|PatchCore|92.1/83.5|93.4/85.2|94.5/86.7|94.8/87.3|94.7/88.0|94.6/87.3|94.4/86.8|93.4/85.6|\n|FeatureNorm|92.0/85.0|92.9/86.5|93.6/87.6|94.0/88.1|93.8/88.1|93.7/88.1|93.4/87.7|92.7/86.4|\n\nIt can be seen that the results significantly decrease with $N_r$ set to 32 and 64. This may be because too few learnable references cannot effectively cover key normal feature patterns. We also further conducted experiments with $N_r$ set to 3072 and 4096. When $N_r$ further increases, the results begin to decrease, and an obvious decrease occurs when $N_r$ is 4096. This may be because too many learnable references can lead to difficulties in optimization during training.\n\n**[To W4]** (1) Due to too many experiments (please see Tab.1, with 6 AD methods, 5 datasets, 4 backbones, and with/without our pretrained features, there is a total of $6 \\times 5 \\times 4 \\times 2 = 240$ experiments), we didn't have enough time and resources to use different random seeds for statistical significance. However, we ensured that all experiments were based on the random seed 42. Thus, we eliminated the influence of randomness, ensuring that all result comparisons are fair. In the revision, we will conduct multiple experiments to report the average values and standard deviation.\n\n**We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you**.\n\n**Note: We try our best to compress the characters, but we still cannot reply to all questions in this box. We borrow some space from the rebuttal box (lower part) of Reviewer rLun to reply to the remaining questions. We sincerely hope that you can take a look**."}]

Author rebuttal:

Meta-review: This paper proposes a pre-training strategy for unsupervised abnormality detection. To this end, the authors proposed to pre-train the backbones with labeled AD datasets with angle- and norm-oriented objectives over residual features, then swaps these representations into standard unsupervised AD methods. Across various datasets and multiple backbones, the swapped-in features consistently improve detection/segmentation metrics.

The paper received generally positive reviews with two borderline accepts, one accept, and one borderline reject. The critical concerns raised by the reviewers include:

- **Misleading terminology:** Describing the overall pipeline as “unsupervised” obscures that pre-training employs the labeled data.
- **Fairness of evaluation:** to show the improvement is not backbone- or recipe-specific, the paper should systematically cover strong modern pretrained reps/backbones (e.g., MAE/DINOv2, EfficientNet/ViT) and report variance/error bars to support reproducibility.
- **Comparison to fine-tuning baselines.** A controlled comparison against per-dataset fine-tuning methods (e.g., FYD, mean-shift contrastive, PANDA) is needed to justify that plug-in representations offer advantages beyond what standard fine-tuning already delivers.
- **Robustness and reference-set sensitivity.** Since residual features depend on few-shot normal references, behavior under contaminated normals and noisy/misaligned reference sets must be analyzed, with a causal explanation for why relative gains persist (or fail) across noise regimes.

During the rebuttal, the authors addressed many of these concerns reasonably, leading some reviewers to increase their scores. While not discussed in the rebuttal, the AC believes the paper misses an important ablation: because both pre-training and evaluation lie in industrial AD, gains could be a trivial consequence of a small domain gap or data overlap. The paper should therefore demonstrate pretrain–test dissimilarity (i.e., no leakage) rather than assume it. This analysis of the pre-training and test data, together with the additional experiments and clarifications presented in the rebuttal, **must** be included in the final version.
Decision: Accept (poster)