ResAD: A Simple Framework for Class Generalizable Anomaly Detection
Authors: Xincheng Yao
Zixin Chen
Chao Gao
Guangtao Zhai
Chongyang Zhang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: class-generalizable anomaly detection
Primary area: machine_vision
TL;DR: we propose a simple but effective class-generalizable AD framework, called ResAD, which can be applied to detect and localize anomalies in new classes.

Abstract: This paper explores the problem of class-generalizable anomaly detection, where the objective is to train one unified AD model that can generalize to detect anomalies in diverse classes from different domains without any retraining or fine-tuning on the target data. Because normal feature representations vary significantly across classes, this will cause the widely studied one-for-one AD models to be poorly classgeneralizable (i.e., performance drops dramatically when used for new classes). In this work, we propose a simple but effective framework (called ResAD) that can be directly applied to detect anomalies in new classes. Our main insight is to learn the residual feature distribution rather than the initial feature distribution. In this way, we can significantly reduce feature variations. Even in new classes, the distribution of normal residual features would not remarkably shift from the learned distribution. Therefore, the learned model can be directly adapted to new classes. ResAD consists of three components: (1) a Feature Converter that converts initial features into residual features; (2) a simple and shallow Feature Constraintor that constrains normal residual features into a spatial hypersphere for further reducing feature variations and maintaining consistency in feature scales among different classes; (3) a Feature Distribution Estimator that estimates the normal residual feature distribution, anomalies can be recognized as out-of-distribution. Despite the simplicity, ResAD can achieve remarkable anomaly detection results when directly used in new classes. The code is available at https://github.com/xcyao00/ResAD.

ResAD: A Simple Framework for Class Generalizable Anomaly Detection
Xincheng Yao', Zixin Chen', Chao Gao", Guangtao Zhai', Chongyang Zhang 1.2+
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University
Moo Key Lab of Artificial Intelligence, AT Institute, Shanghai Jiao Tong University *China Pacific Insurance (Group) Co., Ltd. (i-Dover, CZX15724137864, zhaiguangtao, sunny_ kunny_zhang)@ee 2zaang)ssjtuu eew.jtuuuuu.eee cn' gaochao-027@cpic. com. cn
Abstract
This paper explores the problem of glass-generalizate anomaly detection, where the objective is train unified AD model that can generalize detect anoma- lies diverse classes from different domains without any retraining or fine-tuning on the target data. Because normal feature representations vary significantly across classes, this will cause the widely studied one-for-one AD models to be poorly class- generalizable (i.e., performance drops dramatically when used for new classes). In this work, we propose simple but effective framework (called ResAD) that can be directly applied to detect anomalies in new classes. Our main insight is to learn the residual feature distribution rather than the initial feature distribution. In this way, we can significantly reduce feature variations. Even in new classes, the distribution of normal residual features would not remarkably shift from the learned distribu- tion. Therefore, the learned model can directly adapted new classes. ResAD consists of three components: (1) a Feature Converter that converts initial features into residual features; (2) a simple and shallow Feature Constraintor that constrains normal residual features into a spatial hypersphere for further reducing feature variations and maintaining consistency in feature scales among different classes; (3) Feature Distribution Estimator that estimates the normal residual feature distri- bution, anomalies can be recognized as out-of-distre -istiitribttion. Despite the simplicity, ResAD can achieve remarkable anomaly detection results when directly used in new classes. The code is available at fitttt:://// haaps:://ttttut com/ xcyaa00/Reeaseeoococoeeooaaeea
Introduction
Anomaly detection (AD) has achieved rapid advances in many application domains, such as industrial inspection, video surveillance, and medical lesion detection 9 26). However, applying AD algorithms in real-world scenarios still confronts many challenges. A critical challenge is that there are usually diverse classe and new classes are continually emerging. Most previous one-for one-for-one and also mne-for-many (i.e., learning one AD model for multiple classes) AD methods [13 17 43 [2] 30)
|7|46] 44 are still insufficient to satisfy the requirements of real-world applications. Because such methods still require retraining fine-tuning when encountering new classes, but application users generally don't 't have such ability. Another more fatal point is that some scenarios may not allow retraining target classes due to data privacy issues [1). Therefore, the class- tlass-generalizable ability a critical issue in the AD community, but it still hasn't 't been well studied most AD literatures.
"Corresponding Author.
CClass means the category of the object in the image. For industrial scenarios, it refers to the product category. e.g. bottle, carpet, etc. For medical analysis, it refers to the body organ category, e.g., head CT, retina, etc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training Known Classes
Inference on Diverse New C Classes
Normal
Abnormal
Known C Classes
New Clesses
- Decision Boundary
Selonece
Selennce
Class
New Clee
Natcing
+
Mizsil
Crown Class
Claen
Class
Neew Class
Kriaet Clait
Initaa Feature Distribution
Residual Fenture Distribution
(a) Gonss-aaiaalzzbbee Anomaly Detaction
Res Feature Learning
Figure I: (a): Intuitive illustration of glass-generalizabee anomaly detection (b): Conceptual illustra- tion residual features. The residual feature space has fewer variations compared to the initial feature space. The decision boundary of the residual feature distribution more effectively distinguish anomalies in new classes, rather than treating features of new classes as anomalies.
In this paper, we aim to tackle an academy-valuse and application-requirel task: few-shot class- generalizable anomaly detection, i..., one unified model is trained with samples from multiple known classes, and the goal is that the trained model can generalize to detect anomalies in new classes without any retraining or fine-tuning required. Nonetheless, solving such a task is quite challenging. The current one- creefooronemmmny AD models have almost no ability to directly generalize to new classes. The main challenge is: the normal patterns from different classes are significantly different. This can lead to many normal misdetections of new classes, i.e., normal patches from new classes may be mistaken abnormal as they quite different from the learned normal patterns. Thus, how to design a class geaasss-eeealllbblle AD model under the feature variation circumstance? Our design philosophy is: "seeking invariation from variation". We think that residual features (i.e,, formed by subtracting normal reference features) can be regarded as class-invarian" representations compared to the significantly variant initial features. As shown in (bb) the main merit of normal residual features is: even in new classes, the distribution of normal residual features would not remarkably shift from the learned distribution. Regardless of classes, larger residuals are expected for abnormal features than normal features (please see 3.1.
the target data, only few-shot new class normal samples
To this end, we propose simple but effective class- generalizable AD framework, called ResAD (i.e., Residual Feature Learning based Maass-Generalizabbe Anomaly Detection). ResAD is based on one key insight: residual feature learning, and consists of two key designs: feature hypersphere constraining and feature distribution estimating. First, we propose to use residual features for reducing class feature variations. We employ a pre-trained feature extractor generate normal reference features from few-shot normal reference samples. Each input feature will match the nearest normal reference feature and subtract it to form the residual feature. In this way, the most variable class-related components are very likely to be mutually eliminated, resulting in residual features distributed in a relatively fixed origin-centered region (please see Sec (.1). Second, further reduce the variations in the residual feature space, we take the idea from the one-clas-classificatio (OCC) learning [37 [2 to constrain the feature space. Specifically, we employ a simple and shallow network and propose an abnormal invariant OCC loss to transform normal residual features into a constrained spatial hypersphere. Third, with the hypersphere bypersphere-constrrined feature space, we can easily utilize feature distribution estimator [14] to learn and estimate the normal residual feature distribution, anomalies can be recognized as out-of-distribution. For new classes, as the residual features have fewer variations or are covered by the learned distribution, the whole framework is more class- maasssgenerallzzble.. Our contributions are as follows:
1. To accomplish class- generaaaeell anomaly detection, we propose simple but effective framework: ResAD, which can be applied to detect and localize anomalies in new classes.
2. We are innovatively based on residual feature learning to address the issue of previous one-for- one/many AD methods not being able to generalize to new classes.
We call the classes in training as known classes, others are called as new (or novel, unknown) classes.
Strictly speaking, the residual features not fully invariant, while the variation smaller.

---

3. Comprehensive experiments on six real-world AD datasets are performed to evaluate the AD model's class-generalizable ability. With only 4-shot normal samples as reference, ResAD can achieve remarkable AD results, significantly outperforming the state-of-the-art competing methods.
Related Work
Onny AD Methods. Most AD methods follow the one-for-one/many paradigm. (1) Reconsstraction-basell methods are the most popular AD methods. These methods hold the insight that models trained by normal samples would fail in abnormal image regions. Many previous works
utilize auto-encoders 88 27 42 masked auto-encoders [46], variational auto-encoders [2] and
generative adversarial networks [36 to encode and reconstruct normal data. UniAD [4 is a transformer-based reconstruction model and mainly based on neighbor masked attention address the "identical shortcut" issue to achieve mor- AD. (2) Distillation-basee methods 6 can also be considered as belonging the reconstruction type. These methods train student networks to reconstruct the representation of teacher networks on normal samples, and the assumption is that the student would fail in abnormal features. Recent works mainly focus on feature pyramid [3S 39] reverse distillation [13 38). and asymmetric distillation 33 (3) Embedding--bet bastiddd-aadddd- methods mainly rely on good feature representation and assume that abnormal features are usually far from the normal clusters. Most superior methods
29 30) utilize ImageNet pre-trained networks for feature extraction. However, industrial images generally have an obvious distribution shift from ImageNet. To better account for the distribution shift, subsequent adaptations should be done. The normalizing
flow-based methods [3T 17 32 48 44 are proposed to transform the pre-trained feature distribution into latent Gaussian distribution, and thus can better learn the normal data distribution. HGAD 44 proposes a novel hierarchical Gaussian mixture normalizing flow modeling method to address the "homogeneous mapping" issue for accomplishing one-for-man many AD.
Few-Shot AD Methods. The few-shot AD methods have more similarities with ours. Distance-based approaches such as SPADE []]. PaDiM [12.. and PatchCore [30] can be adapted to address few-shot AD by only making use of few-shot normal samples calculate distance-bar based anomaly scores without training networks. RegAD [8 proposes to train a feature registration network to align imput images and follows PaDiM [2] to model Multivariate Gaussian distribution with few-shot normal samples. The idea in FastRecon [5] is to reconstruct an anomalous sample to its normal version by few-shot support samples. novel regression with distribution regularization is proposed to obtain the optimal transformation from support to query features. Recently, the CLIP-based AD methods, including WinCLIP [99 and VAND [10 show better few-shot AD performance. They both employ a text prompt ensemble strategy to obtain the language-guided anomaly map.
We think class reasragenerll AD and few-shot AD still not the same, they still have some differences. Class-generalizann features of normal samples in the new class as reference. Few-shot AD mainly focuses on how to effectively utilize few-shot normal samples to construct AD models, some dedicated modules may be introduced handle the few-shot normal samples. These methods usually still need re-model in new classes based on few-shot normal samples, e.g., RegAD needs to re-model Multivariate Gaussian distribution for new classes. The CLIP-based methods can be seen as class- haass-enerralizable, as these methods can obtain anomaly maps by aligning vision features with text features. However, they heavily rely on the visual-language comprehension abilities of CLIP and handcrafted text prompts about defects, making them difficult to generalize to anomalies in diverse classes. Compared to the few-shot AD methods, our method can learn a class-generds deaaragneeall AD model, which can be directly applied to new classes only requiring extracting features of few-shot normal samples reference.
AD requires the model to be class-generalizuble, and we only extract
More recently, InCTRL (50) proposes use few-shot normal images as sample prompts and learn to capture in-context residuals between the query image and sample prompts. The idea of in-context residuals in InCTRL is very similar to ours. But our method has obvious differences with InCTRL in the definition and utilization of residuals (please see the detailed differences in Appendix(A.] A.1)
3 Method
Problem Statement. In the dasss-eneeallizable AD task, we focus on the performance of new classes. Formally, let Ierain = n U be training dataset with normal images and some anomalies (i.e.,
3

---

anomalies that exist in training set should also be effectively utilized), where I" = { and I = indicate the collection of normal samples and abnormal samples. As for testing, the model is evaluated on collection of other AD datasets (7 {Ilest .22 _ ...., II except the training dataset. The classes in the test set are drawn from unknown classes C, that are different from the known classes Cs in the training set. Then the goal is to learn one unified model M I - IR that trained with known classes Cj: and can directly adapt to unknown classes C2, without any retraining or fine-tuning the target data (only few-shot (e.g., 4) normal samples reference).
Initial Features
Residual Features
Constrained Features
Test Image
Residual Feature Generation
Feature Extractor
Feature Constraintor
Lace
Distribution Estimator
Few-shot Reference Images
Feature Extractor
Training Testing
Lmm + Log-spp
Reference Feature Pool
Figure 2: Framework overview. Note that the training samples belong to different classes. First, few-shot normal reference samples are fed into a pre-trained Feature Extractor obtain normal reference features. Each initial feature will match the nearest normal reference feature and subtract it form the residual feature. Then, Feature Constraintor utilized to transform the normal residual features into a constrained spatial hypersphere. Finally, we employ a normalizing flow model as the Feature Distribution Estimator to learn and estimate the residual feature distribution.
The proposed ResAD framework is illustrated in Fig The ResAD framework consists of three parts: Feature Extractor, a Feature Constraintor, and Feature Distribution Estimator. These modules will be described below in sequence.
Overview.
3.1 Residual Feature Generating
Residual feature learning is core insight for solving glass-generalizable anomaly detection. In this subsection, we describe how to generate residual features. For any input image 1, E H x3 follow the common practice of previous AD methods to employ pre-trained feature extraction network & to extract features from different levels. Formally, we define L as the total number of levels for use. The feature map from level E (1, L} is denoted where H1, Wi and Oj are the height, width, and channel dimension of the feature map. For an entry a/y.. ø' (I,),, E C level and location (h, w. will match it with the nearest normal reference feature from the corresponding reference feature pool, and then convert it into the residual feature. The details are described in the following:
d' (I.) E IR H, XWW,XC
Reference Feature Pools. The reference feature pools are utilized to store some normal features as reference. For new classes, we will provide few-shot normal samples (i.e., randomly selected and then fixed, please network & will extract hierarchical features for these normal reference images, then the extracted features are sent into the feature pools reference features. For /th level, the Ith reference feature pool composed of P {
discussion sample selection in Appendix (A.)) reference. The pre- trained
un 1h {1,..., Hi), w E (1...., Wi), I {1,..., L), E (1...., N, where denotes the ith normal sample, the N,s is the number of normal reference samples.
Residual Features. For each initial feature "h, we can search the nearest nominal reference feature = argmin || 2//,0 ll2 from the /th reference feature pool Pt. Then, define the residual representation of sh/ w to its closest normal reference feature as:
k x;-
Why can residual features be less sensitive to new classes compared initial features? Because they are obtained by matching and then subtracting, From the principles of representation learning,

---

we know that features of each class generated by well-trained neural networks usually have some class-related attributes the class for distinguishing from other classes [3] The "class-related" means these attributes are typical to the class and distinctive from other classes, representing the most discriminative characteristics of the class. Thus, features from different classes are usually located in different feature domains [40]. However, class-related attributes can also exist in normal reference features (they are usually in the same feature domain as the input query feature), the matching process can be seen as matching the most similar class-related attributes to each query feature. Therefore, by subtracting, the class-related components in the initial features are very likely to be mutually eliminated, leaving the highlighted discrepancy between normals and anomalies (i.e., larger residuals are more likely to be anomalies than normal features). Thus, can be imagined that the normal residual features generally will be distributed in origin-cented centered region, even in new classes, the feature distribution region would not remarkably shift (please see the t-SNE visualization in Fig 3
3.2 Feature Hypersphere Constraining
Even if the feature variations in the residual feature space will be significantly reduced relative to the initial feature space. Features of different classes may still have significant differences in scale, namely, the numerical value scales in the features of different classes may be remarkably different. This can lead to difficulty in obtaining a unified normal- classes, ie., the scales of decision boundaries in different classes may be significantly different, a good decision boundary in one class may be poor in another class. In order to further reduce feature variations and also maintain the consistency in feature scales among different classes, we take the idea from one-class- ree-clas--classficattoo (OCC) learning (34 (22 and propose Feature Constraintor to constrain the initial normal residual features to a spatial hypersphere. The Feature Constraintor Co. !,.." projects the initial residual feature Th to the constrained feature hh,w as Th,w = -C,, (2h,
decision boundary of different
I.l.r
Because we only want to further reduce the variations in the initial residual distribution by constraining and don't 't want to change the distribution overly, we adopt simple network of our Feature Constraintor. complex network may lead to overfitting known features, reducing the generalization ability for new classes (please ablation studies in Tab2
layer as the
Abnormal Invariant OCC Loss. We propose an abnormal invariant OCC loss to optimize our Feature Constraintor. The loss is defined as:
H. Wt
Loce = L H&W; R=1==1
E
(1-1V wh,l)ll h + 1111 + to - s (2)
where v/i.. 1 denotes the (h, w) position on the feature map is anomalous and y/ 0 denotes a normal position (we can downsample the ground-truth mask to a low-resolution mask, which can indicate normal and abnormal positions). The first part in the loss function is pseudo-Huber loss (22. which is used for constraining the normal residual features a hypersphere. However, if we only constrain features to the hypersphere, the network may more easily overfit and simply map all features the hypersphere. If we give the network another objective for anomalous features, this will urge the network to distinguish between normal and abnormal, rather than forming a shortcut solution. Thus, we further introduce an abnormal invariant term by simply predicting the initial features ||=",,, - ==/,,olle l2. "Invariant" means the abnormal residual features remain relatively unchanged relative to themselves and will not be mapped to the hypersphere. In this way, our proposed abnormal invariant OCC loss can not only make the distribution of normal residual features more compact but also keep abnormal residual features as invariant as possible. In addition, by constraining normal features into a hypersphere, the normal feature scales of different classes can also be more consistent. Therefore, after the Feature Constraintor, the normal and abnormal residual features are more distinguishable (see Fig 3 namely, we can obtain better unified decision boundary.
3.3 Feature Distribution Estimating
We employ the normalizing flow (NF) model 14 as our Feature Distribution Estimator to estimate the residual feature distribution. Note that our framework is not limited normalizing flow, and other generative models can also be used as the distribution estimator. Formally, we denote pa,
f.r X' E R + Z E Rº our NF model. The input residual feature 2,,00 will be transformed into
5

---

latent feature 2/,,0 = Po, (2) by the NF model. The estimated residual distribution pe, (a) can be calculated according to the change of variables formula as follows [14] 20
logpe, (x) logoz (z) + l.t./.
where the = Vg= is the Jacobian matrix of the bijective transformation PPe. The model parameters 82 can be optimized by maximizing the log liglliksssooss, and the latent variables Z for normal features are usually assumed to obey (0, I). The maximum likelihood loss function for learning normal residual feature distribution is derived as:
Wi ΣΣ
Hi
(2T) 1 + 2 (=h,,00) 2h, log det.Jh 2
L.l
I
H.W
2
1=1
In the class geseraaaeell AD task, in addition to learning from normal samples, it's also valuable for us to effectively utilize abnormal samples that exist in known classes. Considering that we focus on detecting unknown anomalies in new classes, cannot overfit the anomalies in known classes. Thus, following BGAD [45]. we employ the explicit boundary guided semi-push-puu loss to learn a more discriminative and also generalizable feature distribution estimator. The loss is defined as:
Cbg spp spp
min(logp: - bn,, 0)1 + Σ mmax(logp, bn r, 0)].
(5)
where b, is explicit normal boundary, is margin, Nr, and N₂ denote the number of normal and abnormal features in training batch. We set bn according to the way in BGAD, and T is set to 0.1. Then, the whole loss function for training is as follows:
L = Loce + fmll + Lpg-spp-
In Appendix D we further discuss the sensitivity of balancing among the three loss terms.
3.4 Inference and Anomaly Scoring
For new classes, our method only requires few-shot normal samples to extract features as reference, without any fine-tuning. We feed each test feature z into the Feature Constraintor Ce, and the Feature Distribution Estimator ye, to get the latent feature 2!. The anomaly score is calculated as:
s(x)) = 1 eep ( Ci log(2T)) 2
(x)) logldet.//)
- 2
Then, we upsample all sx.)) in the Ith level to the input image resolution (H x W) using bilinear interpolation and combine all levels (i.e,, sum) obtain the final anomaly map. The maximum score of the anomaly map is taken as the anomaly detection score of the image.
Experiments
4.1 Experimental Setup
Datasets and Metrics. We conduct comprehensive experiments on four real-world industrial AD datasets, including MVTecAD (5). VisA (51). BTAD (25), and MVTecc3 77. The detailed introduction to these datasets is provided in Appendix [ For MVTeec3D, we only use RGB images in the dataset. As for our method's generalizability other domains, we further evaluate our method on medical image dataset, BraTS (24) (for brain tumor segmentation) and a video AD dataset, ShanghaiTech 23] As our method image- we extract video frames in ShanghaiTech as images for use.
Following previous works [5] 6), the anomaly detection performance is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC).
To examine the model's class-generalizan combine the training and test sets of the MVTecAD dataset to train AD methods, and they are subsequently evaluated the test set of other five datasets without any retraining, e.g., train AD
ability, we evaluate the cross-dataset performance. We

---

models on MVTecAD and test on VisA. For MVTecAD, we train AD models on VisA. We report the performance with the number of few-shot normal samples set K 2,4
Implementation Details. All the training and test images are resized and cropped to 224 x 224 resolution. Following the common practice in AD literatures, we utilize the commonly used RideRes- Net50 [4] as the feature extractor, and the outputs from the [1, 2, 3] layers of WideResNet50 are used as the pre-trained features. The parameters of the feature extractor are frozen during training. The layer numbers of the NF model are set as 8. We use the Adam (28) optimizer with weight decay 5e -4 to train the model. The total training epochs are set as 100, and the batch size 32. The learning rate is le 5 initially and dropped by 0.1 after [70, 90] epochs. During training. we randomly select reference samples for each input image to increase residual feature diversity. The network details are in Appendix [ we also evaluate the computation costs of our model and other competing models. We run all the experiments with a single NVIDIA RTX 4090 GPU and random seed 42.
Table I: Anomaly detection and localization results with AUROC metric on six real-world AD datasets under various sew-shot AD settings. ./- means image-level and pixel-Ievel AUROCs, RDAD and UniAD don't t utilize the few-shot normal samples to fine-tune, so the results under 2-shot and 4-shot are the same. For each input image, InCTRL only outputs an image-level anomaly score. Thus, the pixel-level AUROCS of InCTRL are missing.
Raselines
Few-shot AD Methods (Noa- MNo-CCIiD CI.II---sPsl
CI.PPbased AD Methods InCTRL CVPR2224
Setting
Datascts
RDAD CVPR2222
UniAD NeudP52022
SPADE
PaDxM
UNPR2U22
PatchCure
BCCV2022
RegAD
ResAD (Qura)
WinCLIP CVPK2202
ReLAD!
(Durs)
Industrial AD Dutasets
MVTecAD
65.9/719
67.48811
74.6/54.0
79 79.58338
74,3/85.2
60.49333
85.6%4.1
93.1/93.8
91.07-
94,49956
VisA
56.4/99. 9
52.1/8188
71,7/85
68.781.5
65 0.0//044
70.693.
79.5/96.4
81.9944.9
85.8/-
B4.5/55.1
52.7/87.3
67.1/5.66
80,7185.4
88.9995.2
M0./M5.11
N7.295.9
93.697.11
N5.5/55.8
92.3/-
91.1/96.4
MV
58.7900.
51.7/90.4
62.5/78.6
59.684.3
$8.8/83.4
5/666664
64.5855.4
74.1/96.8
78.5:97.5
Average
65.9/52.4
59.654.5
72.4/68.4
74.299.7
89.80030
74.499.2
809/95.8
83,7/95.3
87.1/66.2
2-hbot
Medical AD
ReaT8
49.89667
50.5/885
58.092
49.00
2/33/
$468814
65.78122
55.815
74.60-
67.9/4433
Video
ShanghaiTech
56,2/7.6
55,97994
73,8/7.
70.485..5
7188/87
72.7487.3
78.488.5
78.5/88.1
68.77 7/-
82.4/1.9
All Average
61.6/990
58.954.3
70,2/75.6
69.491.8
68.2/85.6
70.899099
78.293.5
50.8/
83.1/95.2
Industrial Datasets
MVTIcAD
68.977199
67.4/81.1
75.5/64.0
82.5/94.9
50.6/90.2
90.595.7
94.694.2
94.2/969
$6.4/79.9
1/81
75.0655
75.39933
71.7887
73.0
86 8633
84.1995.2
98.897.55
BTAD
$2.7/87.3
67.1/556
$1.7/655
89.9/95.8
$4.0/89 4
90.891.9
95.6997.6
87.2885.8
91.71-
91 01.3/86
MV MV
58 58.790044
51,7/89
62.3/786
62.8/44.5
61.5/87
62.39667
70.987.3
76.00770
59 11-
82.4/7.9
Average
03.992244
39.084.5
73.6/88.4
77.6/94.6
74.5/88.5 5
79.094.9
85.8/57.
85.3/55.6
83.80-
89.7/97.3
4-choo
Medical AD Dataset
BraTS
||
49.8667
59.5/8855
66.39448
60664455
71.295.9
5008873
74.384.
67.3/93.2
76.96-
84.6/66.1
Video AD Datasel
SlanghoiTech
||
56.2/77.6
55.977044
77.1874
74.385.9
778888.2
7647777
7988895
79.888.6
69.30
84.3/92.6
All Axerage
||
61.67700
58.9/84.3
|
73.0760
74.299.2
74 5889
78944
83.095.3
$1.5
81.5 51.51-
88.096.3
Competing Methods. We select the representative oor-for-one AD method (RDAD [3) and the one-for-many AD method (UniAD [47] as baselines. Our method is mainly compared with few- shot AD methods. Following WinCLIIP 19], we adapt three conventional full-shot AD methods, including SPADE [1], PaDiM [12, and PatchCore (30), to the few-shot setting by making use of few-shot normal samples to calculate distance-based anomaly scores. We also compare with the few-shot AD method RegAD (18]. Most of these methods are based on Wide0 extract features. However, these methods still need to re-model in new classes based on few-shot normal samples (see 2e while our ResAD can be directly applied to new classes only requiring extracting features of few-shot normal samples as reference, Then, we also compare with the recent CLIP-based few-shot AD methods, including WinCLIP [19 and InCTRL [50]. To guarantee the rationality of result comparison, we ensure all methods use the same few-shot normal samples, and all results are evaluated based on 224x224 resolution.
4.2 Main Results
Tab represents the comparison results of our ResAD and other SOTA competing methods in image- level AUROC and pixel-level AUROC, respectively, on six real-world AD datasets. Note that all
"No official implementation of WinCLIP is available. We use the public implementation at https://
gi github. wwwpgghaann/ccurtteeeaeae pyttorch
7

---

the results are dataset-level average results across their respective data subsets. Compared to the results on known classes (results in the original papers), the performance of conventional AD methods will drop dramatically when used for new classes, whether it is the one-for-one (RDAD) or the rre-for-many (UniAD) AD method.
By comparison, we can see that our ResAD can significantly outperform all Non-CLIP-basee AD methods on both the 2-shot and 4-shot settings. With more few-shot normal images, the performance of all methods generally becomes better. On average, our ResAD outperforms the best competing model, RegAD, with up to 7.2%/2.9% and 7.6%/2.9% improvements under the 2-shot and 4-shot settings, respectively. In addition, please note that when evaluating RegAD, we utilize the few-shot normal samples to re-model the Multivariate Gaussian distribution for each new class (see See (2). while our ResAD is directly applied to each new class without any re-modeling or fine-tuning. Even with ne-modeling, our method still has advantages over the conventional few-shot AD methods in cross-class detection.
We further implement ResAD model by utilizing the powerful ImageBind [16] as the feature extractor. The outputs from the [8, 16, 24, 32] layers of ImageBind-Hugg are used as the pre-trained features. ImageBind is recently proposed large-scale pre-trained multimodal model, which shows emergent zero-shot and few-shot recognition capabilities across many vision tasks. As shown in Tab by employing model with stronger representation capability, our method can achieve better cross-dataset performance, which significantly outperforms the SOTA CLIP-based AD methods, WinCLIP and InCTRL. This demonstrates that our framework can effectively combine the latest vision models to manifest stronger class- gessseeerrllll ability. What's more, under the 4-shot setting, our ResAD by only using WideResNet50 can achieve comparable or even better results than WinCLII and InCTRL (with more powerful CLIP-based ViT-B/16+), further demonstrating our superiority. Moreover, these two CLIP-based methods also heavily rely on CLIP-based image encoders. When we employ WideResNet50 in these two methods, our method has more advantages than these two methods (please see Appendix Tab
When applied to other domains (medical images and video scenarios). our method also has better cross-domain generalization ability, despite being trained on industrial data (the MVTecAD dataset).
4.3 Ablation Studies
In ablation studies, conduct experiments under the "VisA to MVTecAD" case and use the commonly used WideResNet50 [49] as the feature extractor.
Residual Feature Learning. As shown in Tab without residual feature learning, the cross- dataset performance drops dramatically from 90.5%/99..7% to 72.8%/82.9% This verifies our confirmation that residual feature learning is of vital significance for glass-generalizable anomaly detection. Analogously, any method that can reduce the variations of new class distribution relative to known class distributions is also promising to achieve class-generalizate anomaly detection.
Feature Constraintor. The ablation study on the effectiveness of the Feature Constraintor is also in T a). "w/o Feature Constraintor" means the Loce in E.q.(6) is not used. The effectiveness indicates that by further reducing the variations in the feature distribution and making the distribution of new classes more consistent with the learned distribution, we can achieve better cross-cre class-class/ AD we also present a visualization figure to intuitively show the effect of the Feature
results. In Fig 3
Constraintor.
Abnormal Invariant OCC Loss. The effectiveness of abnormal invariant OCC loss is validated in Ta) "w/o Abnormal Invariant OCC Loss" means the Loce only has the first part of E.q. With the abnormal invariant OCC loss, image-level and pixel-level AUROCS can be improved by 5.6% and 1. 188%, respectively. Moreover, we also find that without this loss, the results would rapidly decrease after certain epochs of training (i.e., overfitting). This shows that keeping abnormal residual features as invariant as possible is beneficial to avoid the Feature Constraintor overfitting and thus achieve better results.
Feature Constraintor Configuration. We further ablate the network architectures of the Feature Constraintor, the results shown in Tbb The results indicate that the simple Cony+
"one-for for-one" means learning one specific AD model for each class, "one-for fore-for-we mor-many" means learning one AD model for multiple classes. However, they both don't consider new classes.
8

---

Table Ablation studies on MVTecAD. (a) "Ours" implementation follows the same configuration as in Tab _ "w/o..." indicates that we remove a certain component relative to "Ours". I-AUROC and P-AUROC mean image-level AUROC and pixel-level AUROC, respectively. (b) "ConvBnRelu" implements simple Conv+BN+ReLA. network. "BasicBlock" adopts the BasicBlock in ResNet. "BottleNeck" adopts the BottleNeck in ResNet. "MultiScaleFusion" is FPN-like architecture to fuse multi-scale features. In
we add BasicBlock/BottlNNcck
after the multi-scale fusion.
(a) Framework ablation studies.
(b) Comparison of different feature constraintors.
Model
I-AUROC
P-AUROC
Network Architectare
F-AUROC
P-AUROC
Ours
90.5
95.7
CansBoRella
90.5
95.7
Residual Learning
72.8
BasicBluck
87.6
94,4
BortleNeck
86.0
94.4
w/o Feature Constraintor
82.3
93.5
MultacaaleRusion
92.9
W/o Abuormal Invariant OCC Loss
84.9
93.9
MultiScalePssion BasicBlack
8233
92.9
MultiScalePssion Baiaraae-nooooonnteeaaaaa BottleNeck
81.0
93.3
deees
(c) Initial Feeture Distribution
Residual Farture Distribution
(c) Initial Resickal Fentures
(d) Constrinee Ras Remictall Featunes
Feature t-SNE visualization. (a) In the initial feature space, the features from different classes are significantly different. (b) In the residual feature space, even the residual feature distribu- tion of unknown classes would not remarkably shift from the known distribution. Note that in (a) and (b), we only show normal residual features and use different colors to represent different classes. (c) The initial residual features. (d) The residual features after the Feature Constraintor.
Figure 3:
network can yield the best performance. We observe a significant performance drop with a more complex feature constraintor (e.g.. Bottleneck, MultiScaleFusion) One possible reason is that a complex network may lead to overfitting, reducing the generalization ability for various anomalies in new classes.
Cross-Class Within One Dataset. We show the results of training with n classes from MVTecAD and testing on the remaining 15- classes. By varying we can demonstrate the sensitivity of the model to different numbers of training classes. Note that different means the number of test classes is different (this will cause the test results of different cannot be compared with each other). Thus, 5 classes as the test classes, including hazelnut, pill, tile, carpet, and zipper. For = 5, the training classes include bottle, cable, capsule, grid, and leather. For n = 10, the training classes include bottle, cable, capsule, grid, leather, metal nut, screw, toothbrush, transistor, and wood. The results under the 4-shot setting are in Tab The results demonstrate that cross-dataset generalization more challenging than cross-class generalization in a single dataset. With more training classes, the results will be better, but the model is not very sensitive.
Table 3: Cross-class results with dif- ferent numbers of training classes n.
=
10
VisA to MVTecAD
96.4797.6 96.8/97.9
95.1/97.2
fixed
4.4 Generalization to Other Anomaly Detection Frameworks
Furthermore, we think that our residual feature learning insight is not limited to the model proposed in this paper, but can be considered as an effective and general method for solving glarrllene anomaly detection. The main reasons are: 1) The process of converting initial features to residual features can be easily applied to other AD models. 2) Residual features are less sensitive to new classes (see Sec 3... In this subsection, we further extend our method to the popular reconstruction-baset AD framework. Specifically, we employ UniAD [47] as baseline and incorporate our method into As UniAD is feature- based AD method, combining our residual feature learning with it is straightforward

---

Table 4: Anomaly detection and localization re- sults when incorporating our method into UniAD, "RFL" represents residual feature learning.
We can convert the initial features into residual features and then perform subsequent feature reconstruction. The experimental results are shown Tab It can be found that the per- formance of UniAD is quite poor when used for new classes, while converting to residual feature learning can significantly improve the model's class-generalizable capacity. The re- markable improvements (e.g.. 25.6% on MVTecAD) validate the effectiveness and generalizability of residual features for designing general- izable AD models.
MVTscAD
VisA
MVTec3D
UniAD +7
17.4/81.1
32.1811 8
67.1/85.6
51.7/89.4
RFL (Ours)
93.0994.9
72.7/86.1
87.3/94.0
76.7/96.9
+25.6/13.8
+20.4/3.3
+20.0/8.4
+25,07.0
4.5 Visualization and Qualitative Results
Visualization Results. Fig 3 a) and (b) show the t- SNE visualization of initial features and residual fea- tures. It can be found that in the initial feature space, the feature distribution new classes is significantly different from the distribution of known classes, re- sulting in poor adaptability of AD models to new classes. However, the variations between different classes can be significantly reduced by converting into residual space. this way, the model's general- izability to new classes can be effectively improved. Fig () and (d) show the i-SNE visualization of ini- tial residual features and residual features after the Feature Constraintor. Results show that the Feature Constraintor can make the normal residual features more compact and more separated from the abnormal features.
Absormal
Greed
MYLLPP
C
Figure 4: Qualitative results. The anomaly score maps are generated under the "VisA to MVTecAD" case.
Qualitative Results. 4 shows qualitative results under the "VisA to MVTecAD" with WideResNet50 as the feature extractor. It can be seen that most SOTA methods fail to generate good anomaly localization maps for new classes, mainly existing many false positives normal regions. However, our method can effectively avoid false positives in normal regions and locate anomalies more accurately. More qualitative results are in Appendix Fig
Conclusion
In this paper, we propose simple but effective framework: ResAD, for achieving class-generalizable anomaly detection. ResAD consists of several simple neural network modules that are easy train and apply in real-world scenarios. Despite the simplicity, ResAD achieves remarkable anomaly detection results new classes. We conclude our findings for future research: residual features are really effective for designing generalizable AD models, and our feature constraining insight also has good reference values for future work.
Limitations. The limitations of our method are discussed in Appendix
Social Impacts. As unified model for class- glarraeeneal anomaly detection, the proposed method does not suffer from particular ethical concerns or negative social impacts. All datasets used are public. All qualitative visualizations are based on industrial product images, which does not infringe personal privacy.
Acknowledgments
This work was supported in part by the National Natural Science Fund of China (62371295), the Shanghai Municipal Science and Technology Major Project (2021SH7277X0102), and the Science and Technology Commission of Shanghai Municipality (22DZ2229005).
10

---

References
[1] Samet Akcay, Amir Atapour-Abarghous and Toby P. Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In ACCV, page
2018.
[2] Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided image generation with conditional invertible neural networks, arXiv preprins arXiv: 1907.02392, 2019.
[3] Kourosh Teimouri Baghaci, Amirreza Payandeh, Pooya Fayyazsanavi, Shahram Rahimi, Zhiqian Chen, and Somaych Bakhtiari Ramezani. Deep representation learning: Fundamentals, perspec- tives, applications, and open challenges. arXiv preprint vrXiv:2211.1473 2022.
[4] Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. arXiv preprint arXiv: 2002.10445, 2020.
[5] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec compre- hensive real-world dataset for unsupervised anomaly detection. In CVPR, 2019.
[6] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In CVPR, 2020.
[7] Paul Bergmann, Xin Jin, David Sattlegger, and Carsten Steger. The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. arXiv preprint arXiv:2112.00045, 2021.
[8] Paul Bergmann, Sindy Lowe, Michael Fauser, David Sattlegger, and Carsten Steger. Improv- ing unsupervised defect segmentation by applying structural similarity to autoencoders. In International Conference Computational Vision Technologies and Applications, 2019.
[9] Yunkang Cao, Xiaohao Xu, Jiangning Zhang, Yugi Cheng, Xiaonan Huang, Guansong Pang, and Weiming Shen. A survey visual anomaly detection: Chanllenge, approach, and prospect. arXiv preprint arXiv:2401. 16402, 2024.
[10] Xuhai Chen, Yue han, and Jiangning Zhang. A zero-/few-shot anomaly classification and segmentation method for cvpr 2023 vnad workshop challenge tracks 1&2: 1st place on zero- shot ad and 4th place on few-shot ad. arXiv preprint 1rXhv:2005.77382, 2023.
[11] Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspon- dences. arXiv preprint arXiv: 2005.02357v3, 2020.
[12] Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: a patch distribution modeling framework for anomaly detection and localization. In 1st International Workshop on Industrial Machine Learning, 2021.
[13] Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embed- ding. In CVPR, 2022.
[14] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
[15) Zheng Fang, Xiaoyang Wang, Haocheng Li, Jiejie Liu, Qiugui Hu, and Jimin Xiao. Fastrecon: Few-shot industrial anomaly detection via fast feature reconstruction. In ICCV, 2023.
[16] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
[17] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In IEEE Winter Confer- ence Application of Computer Vision, 2022.
[18] Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yanfeng Wang. Registration based few-shot anomaly detection. In ECCV, 2022.

---

[19] Jongheon Jeong. Yang Zou, Taewan Kim Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero-/few-shot anomaly classification and segmentation. In CVPR, 2023.
[20] Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu- tions. In Conference and Workshop on Neural Information Processing Systems, 2019.
[21] Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard J. Radke, and Octavia Camps. Towards visually explaining variational autoencoders. In CVPR, 2020.
[22] Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Muller. Explainable deep one-class classification. International Conference on Learning Representations, 2021.
[23] Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked rnn framework. ICCV, 2017.
[24] Bjoern H. Menze, András Jakab, Stefan Bauer, Jayashree Kalpathy-Cramen Keyvan Farahani, Justin S. Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Levente Lanczi, Elizabeth R. Gerstner, Marc-André Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Pa- tricia Buendia, D. Louis Collins, Nicolas Cordier, Jason J. Corso, Antonio Criminisi, Tilak Das, Herve Delingette, Çagatay Demiralp, Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia. Ben Glocker, Polina Golland, Xiaotao Guo, Andac Hamamei, Khan M. Iftekharuddin, Raj Jena, Nigel M. John, Ender Konukoglu, Da- nial Lashkari, José Antonio Mariz, Raphael Meier, Sérgio Pereira, Doina Precup, Stephen J. Price, Tammy Riklin Raviv, Syed M. S. Reza, Michael T. Ryan, Duygu Sarikaya, Lawrence H. Schwartz, Hoo-Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno J. Sousa, Nagesh K. Sub- banna, Gábor Szěkely, Thomas J. Taylor, Owen M. Thomas, Nicholas J. Tustison, Gözde B. Ünal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, and Koen Van Leemput. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging, 2015.
[25] Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. Vt-adl: A vision transformer network for image anomaly detection and localization. arXiv preprint rrXiv:2104.10036 2021.
[26] Guansong Pang. Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 2021.
[27] Hyunjong Park. Jongyoun Noh, and Bumsub Ham. Learning memory memory-guided normality for anomaly detection. In CVPR, 2020.
[28] Diederik P.Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[29] Oliver Rippel, Patrick Mertens, and Dorit Merhof. Modeling the distribution of normal data in pre-trained deep features for anomaly detection. arXiv preprint arXiv:
2020.
[30] Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Scholkopf. Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In CVPR, 2022.
[31] Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semi- supervised defect detection with normalizing flows. IEEE Winter Conference on Application of Computer Vision 2021.
[32] Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Fully convolutional cross-scale-flows for image-based defect detection. In IEEE Winter Conference on Application of Computer Vision 2022.
[33] Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Asymmetric student- teacher networks for industrial anomaly detection. In WACV, 2023.
[34] Lukas Ruff, Robert A. Vandermeulen, Nico Gornitz, Lucas Deecke, and Shoaib Siddiqui. Deep one-class classification. In International Conference on Machine Learning, 2021.
12

---

[35] Mohammadreza Salehi, Niousha Sadjadi, Soroos Hossein Rohban, and Hamid R.Rabice. Mul- tiresolution knowledge distillation for anomaly detection. In CVPR, 2021.
[36] Thomas Schlegl, Philipp Seeb" Seeb"ock, Sebastian M. Waldstein, Ursula Schmidt-Erfun and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Interationnal Conference on Information Processing in Medical Imaging, 2017.
[37] David M.J. Tax and Robert P.W. Duin. Support vector data description. In Machine Learning, pages 45-66, 2004.
[38] Tran Dinh Tien, Anh Tuan Nguyen, Nguyen Hoang Tran, Ta Duc Huy, Soan T.M Duong, Chanh D. Tr. Nguyen, and Steven Q.H. Truong. Revisting reverse distillation for anomaly detection. In CVPR, 2023.
[39] Guodong Wang. Shumin Han, Errui Ding, and Di Huang. Student-teacher feature pyramid matching for unsupervised anomaly detection. In British Machine Vision Conference, 2021.
[40] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to unseen domains: A survey on domain generalization. arXiv preprint arXiv:2103.03097 2021.
[41] Heng Xu, Tianqing Zhu, Lefeng Zhang. Wanlei Zhou, and Philip S Yu. Machine unlearning: A survey, ACM Computing Surveys, 2023.
[42] Jie Yang, Yong Shi, and Zhiquan Qi. Dfr: Deep feature reconstruction for unsupervised anomaly segmentation. arXiv preprint arXiv: 2012.07122, 2020.
[43] Xincheng Yao, Ruoqi Li, Zefeng Qian, Yan Luo, and Chongyang Zhang. Focus the discrepancy: Intra- and inter-correlation learning for image anomaly detection. In ICCV, 2023.
[44] Xincheng Yao, Ruoqi Li, Zefeng Qian, Lu Wang, and Chongyang Zhang. Hierarchical gaussian mixture normalizing flow modeling for unified anomaly detection. In ECCV, 2024.
[45] Xincheng Yao, Ruoqi Li, Jing Zhang, Jun Sun, and Chongyang Zhang. Explicit boundary guided semi-push-pull contrastive learning for supervised anomaly detection. In CVPR, 2023.
[46] Xincheng Yao, Chongyang Zhang, Ruoqi Li, Jun Sun, and Zhenyu Liu, One-for-all: Proposal masked cross-class anomaly detection. In AAAI, 2023.
[47] Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang. Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. arXiv preprint MxXiv:2206003687 2022.
[48] Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows. arXiv preprint arXiv:2111.07677,220 2021.
[49] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[50] Jiawen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learning with shw- sample prompts. In CVPR, 2024.
[51] Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the- difference self-supervised pre-training for anomaly detection and segmentation. In ECCV, 2022.
13

---

Appendix
A More Discussions
A.1 Detailed Comparison with InCTRL
We should have proposed the idea of residual learning independently of InCTRL and almost at the same time (i.e., we completed the initial version of our method during CVPR 2024). But our method has obvious differences with InCTRL in the definition and utilization of residuals. This (i.e. two independent works almost simultaneously proposed the residual learning idea) also demonstrates residual learning is an effective way to achieve class- generalziall anomaly detection. The main differences between our method and InCTRL (50] are as follows:
(I) The definition of residuals in InCTRL is based on feature distances. The residual map is defined by (1E.q-11) in the InCTRL paper): (i j) = 1 (T, f), h(T (i)))))) where (()) returns the embedding of the patch token that most similar Ty (i. among all image patches in Pp, and (-) is the cosine similarity function. Thus, InCTRL is based on residual distance maps, while our method is based on residual features.
By comparison, we think that residual distances in InCTRL can limit the range of residual representa- tion (as the cosine similarity is in [-1,1])- This is not beneficial for distinguishing between normal and abnormal regions, as a position on the residual map is only represented by residual distance value. Within a limited representation range (I-[-1,1] + [0,2]), normal and abnormal residual distance values are more likely to be not strictly separable. Thus, for position on the residual map, it's hard for to make decision based on a scalar value. So, InCTRL makes image-level classification based on whole residual map (see the following (2)). In contrast, our residual features don't 't limit the range of residual representation and can retain the feature properties. In high sigh-dimessional feature space, we can also establish better decision boundaries between normal and abnormal (a basic idea in machine learning: solving di-dmensiionll inseparability by converting to nigh-dimension).
(2) InCTRL devises a holistic anomaly scoring function & to learn the residual distance map M2 = -E_M,, M2 and convert it
anomaly score: s(x) 6(M;; O2) asp(x) (E.q.(8) in the InCTRL paper), where M M2 @ sx(x)) @ sa(xr) (E).. st(x)) is anomaly score based on an image-level residual map Fz. (see E.q.(4) in the InCTRL paper) and 8a (xr) is a text prompt-based anomaly score. Thus, InCTRL is to train binary classification network based on residual distance maps. For each input image, InCTRL finally only outputs an image-level anomaly score. Our method to learn the distribution of residual features, an anomaly score can be estimated for each feature thus can be used to locate anomalies.
(3) Due the designs in InCTRL that we mentioned above, one main advantage of our method is that it can achieve image-level anomaly detection and also pixel-level anomaly localization, while InCTRL only achieves image-level anomaly detection functionality.
As for performance, the average results on six AD datasets of our method are better than InCTRL's (please see Tab [ in the main text).
A.2 Discussion Few-Shot Normal Sample Selection Strategy
In our paper, the few-shot normal reference samples are randomly selected and fixed. This will raise concerns about whether random selection is reasonable and whether it may lead to insufficient representativeness of the reference feature pools. From the perspective of method comparison, we think that random selection is feasible, as long as we ensure that all methods use the same reference samples, the result comparison is reasonable. However, when the difference between normal images too large, it may cause the reference feature pools are not representative. Nonetheless, please further note that, in our method, we only extract the features of the few-shot normal samples and store all the features in the reference feature pools. The reference feature pools don't impair or lose any representation features. For the few-shot normal samples, the reference feature pools are representative enough to them. Therefore, whether the representaaivennes sufficient is determined by the few-shot normal samples themselves rather than our method. For some classes, the few-shot normal samples are representative, while for some hard classes, they may mot be representative enough.
14

---

For practical upplications, this issue should be particularly focused and reasonably addressed. We expect that the reference samples can fully represent their class, it's best have sufficient differences between the reference samples. Thus, the sample selection strategy cannot be random. Of course, the simplest resolution increase the number of reference samples. This is feasible, as in practical applications, the number of reference samples is usually not as strict as the 2-shot and 4-shot our paper.
feasible method is to first cluster all available normal samples into different clusters based on clustering algorithm (e.g., KMeans). Then, based on the number of reference samples, we evenly distribute it to each cluster. When selecting from a cluster, we can prioritize selecting samples closer the center. During clustering. we think that the FID and LPIPS metries are good ways to calculate the difference between two samples. In addition, when there are a large number of reference samples, we can also use the method in PatchCore 30 to select coreset features as reference features, which will be more efficient and also representative.
A.3 Feature Constraintor and Feature Distribution Estimator
The goal of our Feature Constraintor is to constrain initial residual features to spatial hypersphere for further reducing feature variations. After the Feature Constraintor, feature variations can effectively be reduced, but this does not mean that the feature distribution is fixed within the hypersphere. The ideal situation that even in new classes, normal feature distribution fixed within hypersphere, while all anomalous features are outside the hypersphere. Then, only the Feature Constraintor part enough achieve good AD results. However, in practical optimization, it's hard achieve the ideal situation. After the Feature Constraintor, normal and abnormal features may still not be fully separable based on the distances from the features to the center. Therefore, the Feature Distribution Estimator (namely the normalizing flow model used in our method) is used to learn the feature distribution, which can assist us better distinguishing normal and abnormal features.
B Model Architecture and Complexity
Normalizing Flow Model Architecture. The normalizing flow model is mainly based on Real-NVP
[4] architecture, which is composed of the so-called coupling layers. All coupling layers have the same architecture, where a learnable subnet utilized to predict the affine parameters [14]. The convolutional subnet in Real-NVP is replaced with two-layer MLP network. Each coupling layer is followed by random and fixed soft permutation of channels (2 and fixed scaling by constant, similar to ActNorm layers introduced by (20). Furthermore,
adopt the soft clamping of multiplication coefficients used by [1]. Following [7], we add positional embeddings to each coupling layer, which are concatenated with the first half of the input features. The dimension of all positional embeddings is set to 256.
Complexity Comparison. With the image size fixed as 224 224, we compare the number of parameters and per-image inference time with all competitors. We conclude that the advantage of ResAD does not come from a larger model capacity.
Table 5: Complexity comparison between our ResAD and other competing methods.
RDAD
UniAD
SPADE
PaDiM
PutchCore
RegAD
WinCLIP
InCTRL
ResAD
ResADt
Parameters(M1
150.6
6.3
74.5
686.9
69.5
25.2
165.9
117.5
59.2
442.6
Infer time(fps)
5.6
24.4
14.1
21.5
20.2
0.53
21.3
18.8
C Limitations
In this paper, we propose simple but effective AD framework, ResAD, to accomplish class- generalizable anomaly detection. Even if our method manifests good AD performance on six real-world industrial AD datasets, there are still some limitations of our work. One limitation of our work is that we only conducted experiments on data of image modality, it's very valuable to further extend our method to other application domains and data modalities, such as video data and time series, to more comprchensively validate our method's generalizability Our future work will focus
15

---

on further generalizing our method to other data modalities, not only to achieve class- generalizable but also domain-ge ggneraiiaae anomaly detection. Another valuable future work is to incorporate our method into recent SOTA AD methods for achieving hetter class-generalizzale AD performance. In 4.4 we incorporate our method into UniAD and gain remarkable improvements. How to upgrade the other types of anomaly detection methods to class-genens generalizable AD methods and how to find general approach for raasageneral (or even domain domain-generalinn generalizable) anomaly detection will be the future works.
D Datasets
MVTecAD, The MVTecAD (5] dataset is widely used as a standard benchmark for evaluating unsupervised anomaly detection methods. This dataset contains 5354 high-resolution images (3629 images for training and 1725 images for testing) of 15 different product categories. 5 classes consist of textures and the other 10 classes contain objects. A total of 73 different defect types are presented and almost 1900 defective regions are manually annotated in this dataset.
BTAD. The Bean Tech Anomaly Detection dataset [25] is an another popular benchmark, which contains 2830 real-world images of industrial products. Product 1, 2, and 3 of this dataset contain 400, 1000, and 399 training images respectively.
MVTec3D. The MVTec3D 77 dataset is for 3D anomaly detection, which contains 4147 high- resolution 3D point cloud scans paired with 2D RGB images from 10 real-world categories. this dataset, most anomalies can also be detected only through RGB images. Since we focus on image anomaly detection, we only use RGB images of the MVTec3D dataset.
VisA. The Visual Anomaly dataset [31] is larger anomaly detection dataset compared to MVTecAD [5]. This dataset contains 10821 images with 9621 normal and 1200 anomalous samples. In addition to images that only contain single instance, the VisA dataset also have images that contain multiple instances. Moreover, some product categories of the VisA dataset, such as Cashew Chewing gum, Fryum and Pipe fryum, have objects that are roughly aligned. These characteristics make the VisA dataset more challenging than the MVTecAD dataset, whose images only have single instance and are better aligned.
E Sensitivity of Balancing The Loss Terms
During training, we found that summing up the three loss terms (see E.q. and then backpropa- gating gradients to optimize the whole model would lead to unstable training. Then, we used the "torch.detach(]" J" method in the Pytorch library to detach the features after the Feature Constraintor and then sent the detached features into the normalizing flow (NF) model. This simple way can make the model training more stable. Thus, the weight of Loce can be set as 1 (i.e. we can not need to balance Loce with Lul and Lig-sppa spp. as the Feature Constraintor and the NF model parts are separated in the gradient graph). When training the NF model, the Lmi is the basic loss. Thus, we keep the weight of fmm as and set variable À as the weight of Lhg-spp By varying different À values, the results (under the 4-shot setting, from VisA to MVTecAD) about the sensitivity are in Tab
Table 6: Anomaly detection and localization results when varying different values for balancing the three loss terms.
0.1
0.5
2
3
5
10
1/949
90.0/95.6
90.5/95.7
90.6/96.0
89.8/95.3
89.3/95.2
88. 88.7/94.9
Both small and large values can lead to performance degradation. Lip- is to assist model in learning abnormal residual features. Small X may cause the impact of abnormal features on the whole loss (F.q. (b) to be relatively small. Large À may lead to overfitting known anomalies, which not conducive generalization.
16

---

F
Additional Results
Extra Few-shot AD Results. In the main text, the WinCLIP and InCTRL utilize the CLIP-based ViT-B/16+ as the feature extractor. We further employ the WideReset50, which is commonly used in anomaly detection, as the feature extractor in these methods. We note that the two methods do not necessarily need CLIP-based vision encoders. We can remove the vision-language alignment part in two methods and the remaining modules can also achieve anomaly detection. For example, we can send image patches provided by WinCLIP's window mechanism into WideResNet500 and also obtain the window embedding maps different scales as shown in Figure 4 of the WinCLIP paper. However, because the features of WideResNet50
not aligned with the text features, remove the language-gune guguddd anomaly score map and only generate the vision-based anomaly score map based on the few-shot normal samples (the WinCLIP+ in the WinCLIP paper). The results under the 4-shot setting are 7 The results show that our method has more significant superiorities on networks with weaker representation capability. Thus, compared WinCLIP and InCTRL, method is less reliant on the representation capability of the backbone network and more widespreadly applicable for various backbones.
Table Anomaly detection and localization results with WideResNet50 as the feature extractor.
Dataset MVTecAD
WinCLIP
InCTRL
ResAD
86.6/91.6
86.9/-
90,5/95.7
VisA
82.3/
86.2977.4
BTAD
87.7/93.7
90.4/-
95.6/97.6
MVTec3D
63.1/91.7
63.2/-
70.9/97.3
Additional Results on Other Data Groups. the main text, the results are evaluated on a single group of few-shot reference samples. However, the selection of few-shot reference samples may affect the performance the model. To fully represent our model's robustness, we further randomly select two groups of few-shot reference samples. The results under the 4-shot setting are in Tab
Table 8: Anomaly detection and localization results on other two groups of few-shot reference samples.
Dataset
Group1
Group2
Results in Tab
Mcan
MVTecAD
91.0/96.0
90.7/95.9
90.5/95.7
90.7#0.21995.440..
VisA
86.3/97.5
86.9/97.6
86.2/97.4
86.5+0.31/97.5+00000
BTAD
95.3/97.5
95.4/97.6
95.6/97.6
95.410.12/977 610005
MVTec3D
70.2/97.1
70.5/97.3
70.9/97.3
0.5.1...9997...110000
Additional Qualitative Results. We present in Fig additional anomaly localization results of categories from MVTecAD dataset. The anomaly score maps are generated under the "VisA to MVTecAD" case, where AD models trained the VisA dataset.
17

---

Figure 5: Additional qualitative results on MVTecAD.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have summarized contributions well in the abstract and introduction, and the method and experiments sections also reflected these contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please see the implementation details in Sec 4 and network details in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: As mentioned in Abstract, the open-source code will be available at https
//github. com/ wwwp/c/wo.oocccc./ ResAD
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis... cc/
public/guides ottp:////eoooouuubeaesssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wttgg/uddeo/oouuueeeeeeee ssinoolicyy for more details.
//nipscco
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please see Sec 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the large amount of experiments (please see Tab we don't itn't have enough time and resources use different random seeds for multiple experiments. However, we ensured that all experiments were conducted under the same condition of the random seed to 42.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the computation costs Tab
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps///aarrpp..../pppl/////at
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics, and believe that our research conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the Conclusion section, we discussed the potential social impacts of our method.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We think that our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets used in the paper all open-sourced, we have also cited corresponding papers.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper proposes new anomaly detection model, does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "hylI5WBMMe", "reviewer": "Reviewer_fL7E", "summary": "This paper proposes to address cross-class anomaly detection problem. To this end, this study introduce a residual learning framework ResAD. The ResAD framework aims to learning residual feature distribution between target image and reference image. Experiments are conducted to valid the effectiveness of the proposed method.", "questions": "1. The OCC loss is used for constrain the distribution to a fixed region and the NF model is used for estimate the distribution. If the distribution is fixed, what is the meaning of NF model?\n2. There are three loss terms, how about the sensitivities of the balance between the three loss terms.\n3. There is no ablation study to valid the effectiveness of the proposed loss terms which makes the method lack credibility.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The cross-class/class-generalize anomaly detection is a crutial task in the realm of anomaly detection.\n2. The structure of ResAD is simple and effective.", "weaknesses": "1. The idea of residual estimation is highly similar to InCTRL [1].\n2. Lack of comparision with FastRecon[2] and AnomalyGPT[3].\n3. The writing should be improved. The optimization terms are unclear and hard to follow.\n4. In Table.5, there is a reproduced result of WinCLIP on WideResNet50, however the windows in WinCLIP is designed for VIT, how can the authors report the result?\n\n\n[1] Jiawen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learning with few-shot sample prompts. In CVPR, 2024.\n[2] Fang Z, Wang X, Li H, et al. Fastrecon: Few-shot industrial anomaly detection via fast feature reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 17481-17490.\n[3] Gu Z, Zhu B, Zhu G, et al. Anomalygpt: Detecting industrial anomalies using large vision-language models[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(3): 1932-1940.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[To W1].** Thanks for your professional review. We think that our work and InCTRL should be concurrent work. We also initially submitted our work to CVPR2024, and received two weak accepts and one reject (you can see the relevant materials in the rebuttal pdf file). Regretfully, our work was rejected at that time. So, we should have proposed the idea of residual learning independently of InCTRL and almost at the same time. But our method has obvious differences with InCTRL in the definition and utilization of residuals. This (*i.e.*, two independent works almost simultaneously proposed the residual learning idea) also demonstrates residual learning is an effective way to achieve class-generalizable anomaly detection. Then, we made comprehensive revisions to our paper based on the reviewers' suggestions. Subsequently, we also noticed the InCTRL paper and felt that our method has advantages compared to it. Thus, we submitted our revised paper to NeurIPS2024. In CVPR2024, our paper was rejected mainly because a reviewer thought our result comparison with unsupervised AD methods was unreasonable and unfair. In this submission, we mainly compare with few-shot AD methods and InCTRL. The main differences between our method and InCTRL are as follows:\n\n(1) The definition of residuals in InCTRL is based on feature distances. The residual map is defined by ((E.q.(1) in the InCTRL paper): $M_x^l(i,j) = 1 - \\left<T_x^l(i,j),h(T_x^l(i,j)|\\mathcal{P}^\\prime)\\right>$, where $h(T_x^l(i,j)|\\mathcal{P}^\\prime)$ returns the embedding of the patch token that is most similar to $T_x^l(i,j)$ among all image patches in $\\mathcal{P}^\\prime$, and $\\left<\\cdot\\right>$ is the cosine similarity function. Thus, InCTRL is based on residual distance maps, while our method is based on residual features.\n\nBy comparison, we think that residual distances in InCTRL can limit the range of residual representation (as the cosine similarity is in [-1,1]). This is not beneficial for distinguishing between normal and abnormal regions, as a position on the residual map is only represented by a residual distance value. Within a limited representation range (1-[-1,1] $\\rightarrow$ [0,2]), normal and abnormal residual distance values are more likely to be not strictly separable. Thus, for a position on the residual map, it's hard for us to make decision based on a scalar value. So, InCTRL makes image-level classification based on a whole residual map (see the following (2)). In contrast, our residual features don't limit the range of residual representation and can retain the feature properties. In high-dimensional feature space, we can also establish better decision boundaries between normal and abnormal (a basic idea in machine learning: solving low-dimensional inseparability by converting to high-dimension).\n\n(2) InCTRL devises a holistic anomaly scoring function $\\phi$ to learn the residual distance map $M_x = \\frac{1}{n}\\sum_{l=1}^{n}M_x^l$ and convert it to an anomaly score: $s(x) = \\phi(M_x^{+};\\Theta_\\phi) + \\alpha s_p(x)$ (E.q.(8) in the InCTRL paper), where $M_x^{+} = M_x \\oplus s_i(x) \\oplus s_a(x)$ (E.q.(7)). $s_i(x)$ is an anomaly score based on an image-level residual map $F_x$ (see E.q.(4) in the InCTRL paper) and $s_a(x)$ is a text prompt-based anomaly score. Thus, InCTRL is to train a binary classification network based on residual distance maps. For each input image, InCTRL finally only outputs an image-level anomaly score. Our method is to learn the distribution of residual features, an anomaly score can be estimated for each feature, thus can be used to locate anomalies. \n\n(3) Due to the designs in InCTRL that we mentioned above, one main advantage of our method is that it can achieve image-level anomaly detection and also pixel-level anomaly localization, while InCTRL only achieves image-level anomaly detection functionality.\n\nAs for performance, the average results on four AD datasets of our method are better than InCTRL's (please see Table 1 in our paper). In our response to Weakness 1 of Reviewer CQaQ, the cross-domain generalization ability of our method is also better.\n\n**[To W2].** We greatly appreciate your suggestion. Under the 4-shot setting, we further reproduce FastRecon and AnomalyGPT by using the same few-shot samples as ours. Based on the official open-source code, we obtain the following results:\n\n\n|    | FastRecon | AnomalyGPT | AnomalyGPT (ViT-B/16+) | InCTRL | ResAD$^{†}$ (Ours) |\n| ----------- | ----------- | ----------- | ----------- |----------- | ----------- |\n| MVTecAD | 92.3/96.2 | 95.0/96.0 | 92.1/95.3 | 94.5/- | 94.2/96.9|\n|VisA | 82.7/94.1 | 91.3/88.8 | 85.4/86.9 | 88.7/- | 90.8/97.5 |\n|BTAD | 90.1/93.4 | 92.0/96.0 | 90.2/94.9 | 91.7/- | 91.5/96.8 |\n|MVTec3D | 66.5/95.2 | 81.7/96.5 | 75.3/96.2 | 69.1/- | 82.4/97.9|\n|**Average**| 82.9/94.7 | 90.0/94.3 | 85.8/93.3 | 85.8/- | 89.7/97.3|\n\nNote that the original image encoder in AnomalyGPT is significantly larger than the ViT-B/16+ used in InCTRL and our ResAD$^{†}$. When AnomalyGPT also uses ViT-B/16+ as the image encoder, our method is more superior to it.\nIn the revision, we will include the above results and also results under the 2-shot setting in our paper. We will also cite FastRecon and AnomalyGPT and add discussions with them.\n\n**[To W3].** Thanks for your suggestion. We will carefully check and modify obscure expressions and unclear concepts to make our paper easier to follow. We will also seriously consider other reviewers' suggestions to further improve the quality of our paper.\n\n**[Others].** For Weakness 4 and questions, please see the rebuttal pdf file or the following Comment.\n\nIf you still have any questions, we are very glad to further discuss with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sF2UmSZffW", "reviewer": "Reviewer_CQaQ", "summary": "This paper proposed a simple yet effective framework ResAD for class-generalizable anomaly detection by leveraging residual feature learning and a hypersphere constraint. The framework's ability to generalize to new classes without retraining or fine-tuning makes it valuable for real-world applications, providing significant improvements over existing methods. Comprehensive experiments on four real-world industrial AD datasets (MVTecAD, VisA, BTAD, and MVTec3D) demonstrate ResAD's superior performance.", "questions": "(1)If the few-shot reference samples contain anomalies,  will it impact the overall performance a lot?\n\n(2)The way to combine residual feature learning with existing models is not explicitly defined.", "limitations": "Limitations are discussed in Appendix Sec. B.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1)ResAD effectively addresses the challenge of class-generalizable anomaly detection, the generalization ability using only a few normal samples as references makes it highly practical for real-world applications.\n\n(2)The use of residual feature learning to reduce feature variations and improve generalizability is novel and effective\n\n(3)The approach is shown to be robust across different datasets and settings.", "weaknesses": "(1)The experiments are primarily conducted on industrial anomaly detection datasets. While these are relevant, the method's generalizability to other domains, such as medical images or video data, is not fully explored.\n\n(2)The selection of few-shot reference samples may impact performance. Previous methods typically run multiple independent runs using different random seeds to ensure robustness. However, this work only provides results from a single group of samples, which may not fully represent the model's performance variability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[To W1].** We greatly appreciate your suggestion. Under the 4-shot setting, we further evaluate our method on a medical image dataset, BraTS (for brain tumor segmentation) and a video AD dataset, ShanghaiTech (as our method is image-based, we extract video frames as images for use). The comparison results are as follows:\n\n| | RDAD | UniAD | SPADE | PaDiM | PatchCore | RegAD | ResAD | WinCLIP | InCTRL | ResAD$^{†}$|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|BraTS |50.2/58.7 | 57.3/85.1 | 67.1/93.6 | 63.4/93.1 | 70.5/94.6 | 62.1/89.4 | 74.7/94.0 | 68.9/93.5 | 76.9/- | 84.6/96.1|\n|ShanghaiTech|56.2/77.6 | 55.9/79.4 | 77.1/87.4 | 74.3/85.9 | 77.8/88.2 | 76.4/87.7 | 79.8/89.5 | 79.6/88.6 | 69.2/- | 84.3/92.6|\n\nThe results show that when applied to medical images and video scenarios, the cross-domain generalization ability of our method is also good. In the revision, we will include the above results and also results under the 2-shot setting in our paper. We will also attempt more datasets and further discuss our method's generalizability to other domains.\n\n**[To W2].** We greatly appreciate your suggestion. Although we only used a single group of few-shot samples, we strictly ensured that all methods used the same few-shot samples. So, when writing the paper, we thought that the result comparison was reasonable and didn't run on other groups. After reading your suggestion, we think that results on multiple groups are necessary. We then randomly select two groups of few-shot samples and obtain the following results:\n\n| | Group1 | Group2 | Results in paper | Mean$\\pm$Std|\n| --- | --- | --- | --- | --- | \n|MVTecAD | 91.0/96.0 | 90.7/95.9 | 90.5/95.7 | 90.7$\\pm$0.21/95.9$\\pm$0.12|\n|VisA | 86.3/97.5 | 86.9/97.6 | 86.2/97.4 | 86.5$\\pm$0.31/97.5$\\pm$0.08|\n|BTAD | 95.3/97.5 | 95.4/97.6 | 95.6/97.6 | 95.4$\\pm$0.12/97.6$\\pm$0.05|\n|MVTec3D | 70.2/97.1 | 70.5/97.3 | 70.9/97.3 | 70.5$\\pm$0.29/97.2$\\pm$0.09|\n\nIn the revision, we will include the above results and also results under the 2-shot setting in our paper. Due to too many experiments, we currently do not have enough time to provide the results of other methods on these two groups. We will further supply these results in the revised version.\n\n\n**[To Q1].** We think that it will have an impact on the performance. Because it may cause some abnormal features to match similar abnormal features from the reference samples, making these abnormal residual features hard to distinguish from normal residual features. Since the few-shot samples are used as reference, it should be natural not to contain anomalies. For real-world applications, it's also not very hard for us to ensure that the few-shot reference samples are all normal.\n\n**[To Q2].** As UniAD and RDAD are both feature-based AD methods, combining our residual feature learning with them is straightforward. For UniAD, we convert the initial features into residual features and then perform subsequent feature reconstruction. For RDAD, the initial features extracted by the teacher network are converted into residual features as the learning target of the student network. Then, the student network is trained to predict residual representations of the teacher network. In the revision, we will add relevant details to describe clearly how to combine residual feature learning with existing models.\n\nIf you still have any questions, we are very glad to further discuss with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zsxDRUgAkx", "reviewer": "Reviewer_226V", "summary": "This paper proposes a simple but effective framework that can be directly applied to detect anomalies in new classes. The main insight is learning the residual feature distribution rather than the initial one. In this way, we can significantly reduce feature variations. Even in new classes, the distribution of normal residual features would not remarkably shift from the learned distribution. Experiments were conducted on four datasets and achieved remarkable anomaly detection results.", "questions": "1. If the difference between normal images is relatively large, such as the breakfast_box class and screw_bag class in the MVTec Loco AD dataset, how can the reference feature pool be ensured? Intuitively, if the normal image difference is too large and the reference feature pool is not representative, the scheme has the hidden danger of a high false detection rate. If you're not using this dataset for an experiment, you should mention it in the text, which is good for the community.\n2. Is the random selection of normal features in the reference feature pool a good strategy at fixed? Is it better to maximize the difference?\n3. pixel-level AUROCs of InCTRL in Table 1 should be displayed. If not, it should be explained that it was not done by itself rather than that it could not be obtained.\n4. Line 225: As far as I know there are 15 products and their corresponding exceptions in MVTecAD. Did you train the model using 15 product images and test it on various VisA classes? Although MVTecAD and VisA are two different data sets, they are just two sets containing multiple classes. So I think you should show the results of training with n classes from MVTecAD and testing on the remaining 15-n classes, with n as a hyperparameter looking for sensitivity, instead of testing across datasets. It would not be difficult for you to write your experiment in detail, and it would be more convincing if the results of the experiment appeared in the paper.", "limitations": "This paper objectively mentions the limitations of this article, and there is no potential negative impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is original, high quality, clear, and easy to understand. The proposed method has a good heuristic effect on establishing a general anomaly detection model and will become a valuable baseline for the community after the release of the code.", "weaknesses": "1. Although unnecessary, I recommend punctuation at the end of a formula. This is one of the few formatting problems I can pick out. [Well written]\n2. In Figure (b), it is suggested that abnormal should use a triangle icon. The difference between a hexagon and a circle is too small to see clearly.\n3. The large difference between normal images should be considered, and image difference indicators such as FID and LPIPS can be used to calculate the difference inside the normal images in the data set you show. The difference should be relatively small, which is a potential false alarm hazard.\n4. As stated in point 4 of the questions, the experimental setup of training on MVTecAD and then testing on the various classes of VisA is not reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[To W1].** Thanks for your suggestion. We then checked the formula writing in several other papers and found that there is punctuation at the end of a formula. This is a quite good detail suggestion. We will make modifications in the revised version.\n\n**[To W2].** Thanks for your suggestion. Using triangles to represent anomalies does look clearer. We will further improve Figure 1 in the revised version. \n\n**[To W3, Q1, and Q2].** We greatly appreciate your suggestion. Yes, when the difference between normal images is too large, it may cause the reference feature pool is not representative. For practical applications, this issue should be particularly focused and reasonably addressed. Of course, the simplest resolution is to increase the number of reference samples. This is feasible, as in practical applications, the number of reference samples is usually not as strict as the 2-shot and 4-shot (following previous papers) in our paper. From the perspective of method comparison, we think that random selection is ok, as long as we ensure that all methods use the same reference samples, the result comparison is reasonable. **[To Q2]** However, in practical applications, we expect that the reference samples can fully represent their class, so it's best to have sufficient differences between the reference samples. Thus, the sample selection strategy cannot be random. **[To Q1]** A feasible method is to first cluster all available normal samples into different clusters based on a clustering algorithm (*e.g.*, KMeans). Then, based on the number of reference samples, we evenly distribute it to each cluster. When selecting from a cluster, we can prioritize selecting samples closer to the center. During clustering, we think that the FID and LPIPS you mentioned are good ways to calculate the difference between two samples. In addition, when there are a large number of reference samples, we can also use the method in PatchCore to select coreset features as reference features, which will be more efficient and also representative. In the revision, we will add the above discussion on sample selection strategy to the paper.\n\n**[To W4 and Q4].** Yes, we train the model using 15 classes in MVTecAD and test it on 12 classes in VisA. For MVTecAD, we train on 12 classes in VisA and test on 15 classes in MVTecAD. Adopting the cross-dataset experimental setup is because we think that it's more challenging than cross-classes within a single dataset, thus can better verify the model's class generalization ability. A dataset may be collected under the same photography condition, variations in other factors besides the object itself may be minimal. For example, all images in MVTecAD only contain a single object, while the images of some classes in VisA have multiple objects. The backgrounds in MVTec3D are all black, which is not the case in other datasets. In addition, we also follow InCTRL to use the cross-dataset experimental setup. \n\nWe think that your mentioned experimental setting is also very reasonable, as by varying $n$, we can demonstrate the sensitivity of the model to different numbers of training classes. Under the 4-shot setting, we further run our model on MVTecAD (with $n=5, 10$). Note that different $n$ means the number of test classes $15-n$ is different (this will cause the test results of different n cannot be compared with each other). Thus, we use fixed 5 classes as the test classes, including hazelnut, pill, tile, carpet, and zipper. For $n=5$, the training classes include bottle, cable, capsule, grid, and leather. For $n=10$, the training classes include bottle, cable, capsule, grid, leather, metal nut, screw, toothbrush, transistor, and wood. The results are as follows:\n\n|n=5 | n=10 | VisA to MVTecAD |\n| --- | --- | --- | \n| 96.4/97.6 | 96.8/97.9 | 95.1/97.2 |\n\nThe results demonstrate that cross-dataset is more challenging than cross-class in a single dataset. With more training classes, the results will be better, but the model is not very sensitive. Due to too many experiments, we currently do not have enough time to provide the results of other methods under this setting. In the revision, we will also add this setting to our experimental setup and complete other methods' results.\n\n**[To Q3].** For each input image, InCTRL finally only outputs an image-level anomaly score. Thus, InCTRL cannot calculate pixel-level AUROCs. In the revision, we will add a relevant explanation.\n\nIf you still have any questions, we are very glad to further discuss with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2nRg1Funl0", "reviewer": "Reviewer_6bzj", "summary": "The paper analyzes the class-generalizable anomaly detection problem and introduces residual feature learning. \nBased on the residual features, the paper proposes a simple AD framework, i.e., ResAD, which incorporates OCC loss and distribution estimating to distinguish normal and abnormal data.\nThe experimental results demonstrate that the ResAD performs well on real-world industrial AD datasets.", "questions": "1. What is the superiority of the proposed simple subtraction-based residual learning compared with the residual learning in InCTRL?\n2. In the related work, the author claims that the CLIP-based methods are difficult to generalize to anomalies in diverse classes. However, according to the experiment results, the proposed methods only perform well on industrial AD datasets while InCTRL performs well on various types of datasets, including Medical datasets and  Semantic datasets. Why does your method only compare different classes on industrial datasets, instead of comparing against anomaly datasets from other domain?  I am wondering how the ResAD performs on the datasets from other domains.\n3. What are the main advantages of ResAD compared with WinCLIP and InCTRL since the generalization ability and complexity of ResAD are not as good as WinCLIP and InCTRL. \n4. In the residual feature construction process, the residual feature is highly related to the closest normal reference features in the reference feature pool. Are the few-shot reference samples enough to represent the class-related attributes?\n5. In table 1, the authors mention that RDAD and UniAD don't utilize the few-shot normal samples to fine-tune, so the results under 2-shot and 4-shot are the same.  RDAD and UniAD don’t require few-shot normal samples to fine-tune or refer, while the proposed method provide few normal samples to refer, so I believe it is meaningful to compare your method with those that require few-shot normal samples to refer, such as inctrl and winclip. Comparing it with RDAD and UniAD seems to be unfair especially in Table 3 . How do the results of corporating proposed method into WinCLIP and InCTRL?", "limitations": "The authors did not give a discussion on the limitations of the proposed method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper analyzes the few-shot class generalizable anomaly detection problem and delivers an interesting insight into residual features.\n2. The proposed method is intuitive and easy to understand.\n2. The paper is well-written and organized.", "weaknesses": "1. The residual learning for few-shot AD has already been proposed in inCTRL[1]. The proposed Multi-Layer Patch-Level Residual Learning scheme in InCRTL is more sophisticated and reasonable than the direct subtraction in this paper.\n2. The results in Table 1 of InCTRL are not consistent with the results in the original paper. Compared with the original results of InCTRL, the ResAD results do not achieve the SOTA performance. \n3. The paper aims to achieve generalization across different classes. I think the authors should compare the accuracy of each class on the Visa dataset with other methods to demonstrate the generalization capability of your approach for different classes, rather than taking the average accuracy of different classes in the dataset.\n\n[1]Jiawen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learning with few-shot sample prompts. In CVPR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[To W1 and Q1].** Thanks for your professional review. We think that our work and InCTRL should be concurrent work. We also initially submitted our work to CVPR2024, and received two weak accepts and one reject (you can see the relevant materials in the rebuttal pdf file). Regretfully, our work was rejected at that time. So, we should have proposed the idea of residual learning independently of InCTRL and almost at the same time. But our method has obvious differences with InCTRL in the definition and utilization of residuals. In CVPR2024, our paper was rejected mainly because a reviewer thought our result comparison with unsupervised AD methods was unreasonable and unfair. In this submission, we mainly compare with few-shot AD methods and InCTRL. **Please see our response to Reviewer fL7E's Weakness 1, we provide a detailed comparison between our method and InCTRL**. Based on the comparison, we think that our method has the following advantages:\n\n(1) One main advantage of our method is that it can achieve image-level anomaly detection and also pixel-level anomaly localization, while InCTRL only achieves image-level anomaly detection (due to the designs in their method).\n\n(2) Compared to residual distance maps (in InCTRL), residual features are easier to integrate into other feature-based AD methods. For example, in Sec.4.4, we combine the residual feature learning with UniAD and RDAD, which can effectively improve the models’ class-generalizable capacity. In InCTRL, the authors devise an anomaly scoring network to learn the residual distance maps. The residual distance maps seem not easy to integrate into other AD methods (based on our analysis in the response to Reviewer fL7E’s Weakness 1).\n\n**[To W2].** We checked the results again and found that the results of InCTRL we reported were the same as in its original paper (94.0 and 85.5 under the 2-shot setting, 94.5 and 87.7 under the 4-shot setting). In the InCTRL paper, there are two tables, the AUROC results are in Table 1, and Table 2 are the AUPRC results, which are overall higher than the AUROC results. The original results you mentioned may be from Table 2 of the InCTRL paper. In Table 1 of our paper, we report the AUROC results, the average results on four AD datasets of our method are also better than InCTRL’s.\n\n**[To W3].** Thanks for your suggestion. Due to the page limitation and the need to extensively validate the effectiveness of our method on multiple datasets, we adopted to report the dataset-level average results, which was also the way adopted in the InCTRL paper. We present the detailed results on the VisA dataset in the rebuttal pdf file and also the following Comment. \n\n**[To Q2].** When writing the paper, we thought the results on these four industrial AD datasets were enough to validate the effectiveness of our method (In Table 1, our method can achieve better average results on these datasets than other methods), so we didn't consider more datasets from other domains. Please see our response to Weakness 1 of Reviewer CQaQ, we further run experiments on a medical dataset and a video AD dataset.\n\n**[To Q3].** According to our response to Weakness 2, our method is better than InCTRL and WinCLIP in terms of the average results on four AD datasets. The advantages to InCTRL are discussed in **[To W1 and Q1]**. Compared to WinCLIP, we think the advantages are:\n\n(1) WinCLIP generates anomaly score maps by directly calculating the similarity between vision and text features. The model is not trained on AD datasets. Thus, WinCLIP is more likely to rely on the visual-language comprehension abilities of CLIP. When text prompts cannot capture the desired anomaly semantics, the results may be poor. In addition, WinCLIP requires text prompts, which can bring extra complexity, while our method does not require any text.\n\n(2) Due to its sliding window mechanism, WinCLIP has low efficiency. In Table 4 in the Appendix, we provide the number of parameters and per-image inference time of our method and other methods (we list the table in the rebuttal pdf file and also the following Comment). The inference speed of WinCLIP is 0.51fps, while our method's is 18.8fps.\n\n**[To Q4].** Please see our response to Weakness 3 of Reviewer 226V.\n\n**[To Q5].** Yes, in Table 1, our method is mainly compared with these few-shot AD methods. Including the cross-dataset results of UniAD and RDAD is not for comparison, but to demonstrate that conventional one-for-one (RDAD) and also one-for-many (UniAD) AD methods cannot be directly applied to new classes (compared to the results in their original papers). Thus, achieving class-generalizable anomaly detection requires new insights and specific designs. In Table 3, we mainly aim to demonstrate that the residual feature learning can be easily incorporated into conventional feature-based AD methods and can effectively improve their class-generalizable capacity.\n\nWe also considered combining our method with WinCLIP and InCTRL but found it not easy to achieve. In WinCLIP, the model is based on the alignment between vision and text features. As the semantics of residual features and initial features are different, converting to residual features can lead to misalignment with text features. In addition, our method is to learn the residual feature distribution, while WinCLIP doesn't have the training stage on AD datasets. In InCTRL, the model is designed based on residual distance maps. InCTRL devises an anomaly scoring network (discriminative model) to learn the residual distance map and convert it to an anomaly score. Our method is designed based on residual features, which utilizes a normalizing flow model (probabilistic generative model) to learn the residual feature distribution. The obvious differences in the definition and utilization of residuals between our method and InCTRL make integrating with InCTRL also not easy.\n\nIf you still have any questions, we are very glad to further discuss with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are very grateful for all your constructive suggestions. Please see our specific responses to each reviewer. 

In the Author Rebuttal pdf file, we provide some relevant materials. We recommend that Reviewer fL7E and 6bzj can download the pdf file and see the contents in it.

Meta-review: The work tackles a problem called class-generalizable anomaly detection on image data and introduces a new approach called ResAD for the problem. The key idea in the approach is residual feature learning based on a popular VLM backbone, CLIP. Th approach is evaluated on four industrial defect detection datasets and compared with several relevant methods, including two recent CLIP-based methods, WinCLIP and InCTRL.

After the author-reviewer discussion period, all reviews are positive toward this work. Reviewers CQaQ and 226V are inclined to acceptance throughout the review process. Reviewer 6bz had some major doubts on this work in the model design and the experimental comparison. The authors managed to address these concerns properly, resulting in an increase of rating from borderline accept to weak accept. Reviewer fL7E had major concerns over the difference of the work to a recent method InCTRL and comparison to related methods FastRecon and AnomalyGPT. The authors have also adequately addressed these concerns during the author-reviewer discussion period, leading to a rating increase from borderline reject to borderline accept. 

Overall, all reviewers generally appreciate this work in terms of problem setting, technical design, and empirical justification. I therefore recommend accepting the paper to NeurIPS.

I'd like to further recommend the paper as a spotlight paper because i) the studied problem is an emerging important AD setting that uses large pre-trained vision-language models to support class-generalizable AD (or generalist AD), ii) the proposed residual feature learning idea is interesting (despite a similar insight was proposed in InCTRL), and iii) the detection performance is promising.
Decision: Accept (spotlight)