PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection
Authors: Qihang Zhou
Jiangtao Yan
Shibo He
Wenchao Meng
Jiming Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D anomaly detection
Zero-shot 3D anomaly detection
Primary area: machine_vision
TL;DR: This paper for the first time adapts CLIP to comprehend 3D anomalies in a zero-shot manner.

Abstract: Zero-shot (ZS) 3D anomaly detection is a crucial yet unexplored field that addresses scenarios where target 3D training samples are unavailable due to practical concerns like privacy protection. This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD provides a unified framework to comprehend 3D anomalies from both points and pixels. In this framework, PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. To capture the generic anomaly semantics into PointAD, we propose hybrid representation learning that optimizes the learnable text prompts from 3D and 2D through auxiliary point clouds. The collaboration optimization between point and pixel representations jointly facilitates our model to grasp underlying 3D anomaly patterns, contributing to detecting and segmenting anomalies of unseen diverse 3D objects. Through the alignment of 3D and 2D space, our model can directly integrate RGB information, further enhancing the understanding of 3D anomalies in a plug-and-play manner. Extensive experiments show the superiority of PointAD in ZS 3D anomaly detection across diverse unseen objects.

PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection
Qihang Zhou', Jiangtao Yan', Shibo He': Wenchao Jeming Chen' College of Control Science and Engineering, Zhejiang University (zqhang, jtaoy, s18he, wmengzju, cjm}@zju. edu. cn
Abstract
Zero-shot (ZS) 3D anomaly detection is crucial yet unexplored field that ad- dresses scenarios where target 3D training samples unavailable due to practical concerns like privacy protection. This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD provides unified framework to compre- hend 3D anomalies from both points and pixels. this framework, PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. To capture the generic anomaly semantics into PointAD, we propose hybrid representation learning that optimizes the learnable text prompts from 3D and 2D through auxiliary point clouds. The collaboration optimization between point and pixel representations jointly facilitates our model to grasp underlying 3D anomaly patterns, contributing to detecting and segmenting anomalies of unseen diverse 3D objects. Through the aligmment of 3D and 2D space, our model directly integrate RGB information, further enhancing the understanding anomalies in plug-and houg-and-pp and-play manner. Extensive experiments show the superiority PointAD in ZS 3D anomaly detection across diverse unseen objects. Code available htttt::///// giighu........ com htt///gghannPoon..
3D
Introduction
Anomaly detection, significant field within deep learning, has been widely applied to diverse domains, including industrial inspection [2]
32 36
37
44
15
24 55
19
63 While 2D
anomaly detection has been extensively studied by exploring RGB information 23 56 50
51 8 29
real-world anomalies typically present themselves with abnormal spatial characteristics. Relying solely on RGB information poses challenges in detecting some anomalies in many cases, e.g., when the defect mimics the appearance of the object's background or foreground, as shown Figure ((a)
The emerging field of 3D anomaly detection aims to unveil these spatial relations indicative of
abnormal patterns 22
However, current 3D anomaly detection methods typically store normal point features during training and identify anomalies by measuring the distance between the test feature and these stored features [22
[3]. They all depend on the assumption that target point clouds are available and entirely normal. This assumption does hold in various situations when the training samples in the target dataset are inaccessible due to privacy protection (e.g., involvement of trade secrets) or the absence of target training data (e.g., a new product never seen before) 663 Figure depicts the setting discrepancy between ZS 3D and unsupervised anomaly detection. These methods mentioned above, which detect anomalies by memorizing or reconstructing normal point features, have limitations in generalizing to unseen objects in Figure I(c) While zero-shot (ZS) anomaly detection has been explored in 2D images 35 63 7S 3D anomaly detection remains research blank. It is challenging task as 7S
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

20 RGB input
Point cloud input
Ground truih
RGB score map
Point map
Triiiin date
Train
Unsupervised setting
BTF performance
100
Unsuparvised
Some object
specific
Chyeett
Zero-shot
Texi
date
Auxiliory
Train
Zaro-shot sefting
Object- agroshic Model
Different objects
Teat
dateldotelde date
Test
TesT
L-AUROC
AUPRO
(a) Comparison on anomaly segmentation us- (b) Comparison between ZS (c) BTF Performance degra- ing different modalities.
and unsupervised settings. dation on MVTec3D-AD.
Figure 1: Motivation of zero-shot 3D anomaly detection. (a): Top: The hole the cookies presents a similar appearance to the background. Bottom: Surface damage on the potato is unapparent to the object foreground. In these cases, leveraging RGB information makes it difficult to detect anomalies that imitate the color patterns of the background or foreground. However, effective recognition can be achieved by modeling the point relations within corresponding point clouds. (b) and (c) depicts the setting difference of ZS and unsupervised manner.
3D anomaly detection necessitates the model detect 3D anomalies across unseen point clouds with diverse class semantics, requiring a robust generalization capacity in the detection model. Recently, Vision-Language Models (VLMs) with their strong generalization capabilities have been applied to various downstream tasks 40 61
49 25 26). Particularly, CLIP has demonstrated its strong ZS performance detect 2D anomalies [35 24 63 Integrating CLIP into the detection model presents
a potential solution to the challenging yet unexplored ZS 3D anomaly detection.
In this paper, we propose unified framework, namely PointAD, to transfer the knowledge of CLIP to detect 3D anomalies in a ZS manner. PointAD comprehends point clouds from both 3D and 2D: (1) deriving 2D representations of point clouds via CLIP by rendering them from multiple views, (2) understanding 3D representations by projecting 2D representations back to 3D, and (3) enhancing 3D comprehension by additional regularization on 2D representations. After grasping point clouds from points and pixels, we propose hybrid representation learning to capture generic normality and abnormality w.r.t. point and pixel information into learnable text prompts [63]. Specifically, since 3D representation manifests its 2D renderings from different views, we treat each representation as one instance and achieve 3D representation aggregation via multiple instance lcarning (MIL). On this basis, PointAD explicitly aligns the 2D anomalies, rendered from 3D anomalics, to further enhance 3D understanding. We formulate these 2D anomaly recognition tasks from the multi-task learning (MTL) perspective. PointAD collaboratively learns point and pixel representations, promoting the in-depth understanding of underlying abnormal patterns and thus achieving superior ZS normality and abnormality point recognition. Furthermore, benefiting from collaboration optimization, PointAD can directly integrate additional RGB information and perform ZS multimodal 3D (M3D) detection without extra modules and retraining. The main contributions of this paper are summarized as follows:
To the best of our knowledge, we are the first to investigate the challenging yet valuable ZS 3D anomaly detection domain. We propose to transfer the strong recognition generalization of CLIP to detect and segment 3D anomalies over diverse objects.
We introduce novel ZS 3D anomaly detection approach called PointAD, which provides unified framework to understand 3D anomalies from points and pixels. Hybrid representation learning proposed incorporate the generic normality and abnormality semantics into PointAD, enabling a thorough understanding of 3D anomalies.
PointAD can incorporate 2D RGB information in a plug-and aodg--ad--layy manner for testing. In contrast to other methods that require storing RGB information separately, PointAD offers unified framework to perform ZS M3I) anomaly detection directly.
Extensive experiments are conducted demonstrate the superiority of our model in detecting and segmenting 3D anomalies, even outperforming some unsupervised SOTA methods that memorize normal features of target objects in certain metrics. We hope that our model will serve as springboard for future research on ZS 3D and M3D anomaly detection.
2 Related Work
3D Anomaly Detection MVTec3D-AD 4 Eyecandies 5 and Real3D-AD 31 provide the point cloud anomalies and the corresponding 2D-RGB information. MVTec3D-AD bridges the connection between 3D and 2D anomaly detection. 3D-ST 4 uses teacher net extract dense local geometric

---

descriptors and design a student net to match such descriptors, AST 45
introduces asymmetric
teacher and student net further improve 3D anomaly detection. IMRNet 28 and 3DSR (58] detect 3D anomalies by reconstruction errors. Instead of only using point clouds, BTF 222, M3DM S4 53 CPFM 77. and SDM [3] integrate point features and RGB pixel features to detect 3D anomalies. While these approaches exhibit commendable performance by storing object-specific normal point and pixel features within the unsupervised learning framework, such paradigms simultancously limit their generalization capacity to point clouds from unseen objects, which is crucial to detecting anomalies when the target object is unavailable. To the best our knowledge, no solution addresses this valuable yet challenging problem. To fill this gap, introduce PointAD, designed to identify unseen anomalies across diverse objects. PointAD extends CLIP to the realm of ZS 3D anomaly detection and shows robust generalization in capturing generic normality and abnormality within point clouds. Furthermore, PointAD serves as a unified framework, allowing seamless integration of point cloud and RGB modality without additional training.
3D Feature Extraction Conventional methods of 3D feature extraction typically employ a point- based network like PointNet (38 or PointNet+++ to extract 3D features from point clouds. Alternative approaches convert 3D data into a 2D format [48] [8] enabling 2D image backbones to process 3D information. PointCLIP [59] directly projects raw points onto image planes for efficiency, but this approach causes the produced depth map to lack geometric details. Instead, rendering-base based methods [48] 21) generate 2D renderings by rendering point clouds, allowing for better preservation of local semantics. CPFM 7 stores normal features of these 2D renderings for unsupervised 3D anomaly detection. In this paper. we apply this rendering strategy to the source samples to capture generic anomaly semantics for recognizing abnormalities in unseen objects.
Prompt Learning Instead of fine-tuning the whole network, prompt learning just optimizes the model to adapt the network to downstream tasks. CoOp [6 60) introduces global context opti- mization to update learnable text prompts for few-shot recognition. DenseCLIP 41 extends it to the dense classifications. More recently, AnomalyCLIP [63] proposes object-agnostic prompt learning to capture the generic normality and abnormality for images. Our model first introduces hybrid representation learning for ZS 3D anomaly detection, enabling the detection of anomalies and abnormal regions.
3 PointAD
3.1 A Review of CLIP
CLIP, a representative VLM, aligns visual representations to the corresponding textual representations, where an image is classified by comparing the cosine similarity between its visual representation and textual representations of given class-specific text prompts. Specifically, given an image x; and target class set C, visual encoders output the global visual representation Si E IRd and local visual representations f. E Rhxxxd where h, w, and d are the height, width, and dimension, respectively. Textual representations Ir. are encoded by textual encoder T with the commonly used text prompt template A photo of [c], where e E C. The probability of x; belongs can be computed as:
exp(coo ge: f()/T)
P(ge, fi) Ecec (xp(cos (t)))/
(1)
where cos(ss) .) and T represent the cosine similarity and temperature used in CLIP, respectively. The segmentation Site) E khhx for class e can be computed as Seg(ge, f("),, where each entry (u,z) is calculated as P(ge: (T
3.2 Overview of PointAD
ZS 3D anomaly detection requires strong generalization capacity to anomalies unseen objects with diverse object semantics. In this paper, we propose a unified framework, namely PointAD, to detect and segment 3D anomalies in ZS manner. In Figure 2 PointAD understands point clouds from both pixel and point perspectives. To make CLIP understand 3D point clouds, we first render point clouds from multiple views and extract the pixel representations of these generated 2D renderings via the visual encoder of CLIP. And then, we derive point representations by projecting these pixel representations back to 3D. Learning generic normality and abnormality is significant in recognizing across-object anomalies. We propose hybrid representative learning, which focuses on glocal point and pixel abnormality, optimize normality and abnormality text prompts, enabling PointAD with strong generalization to identify 3D anomalies on diverse objects. Benefiting from the hybrid representation learning, PointAD can directly incorporate 2D RGB information during testing to achieve ZS M3D detection.
3

---

MTL--based 3D branch
MTL-based 2D branch
9: abnormal object-aware Sjjct-awwreeaghostiii
learnable text prompts
damaged class
or
hybrid
domaged object
poba
0,98 A
0.92
Text encoder
Textual representation
Text encoder
Global representations
Mutti-view global 20 represeniations
Ve
class
or
Multi view local 2D representations
object
Vislon
encoder
9." object-oware//nee awvore/ognostic
My
learnable text prompts
Local view wise 3D representations
Render(R)
semantics
For local semontics
ReratiooR
Frosen
Leamable
Homittorinn product
Casine simitrity
Loyer-wise.ccom Cagragetion
wise Average
Figure 2: Framework of PointAD. To transfer the strong generalization of CLIP from 2D to 3D, point clouds and corresponding ground truths are respectively rendered into 2D renderings from multi-view. Then, vision encoder of CLIP extracts the renderings to derive 2D global and local representations. These representations are transformed into glocal 3D point representations to learn 3D anomaly semantics within point clouds. Finally, we align the normality and abnormality from both point perspectives (multiple instance learning) and pixel perspectives (multiple task learning) and propose hybrid loss jointly optimize the text embeddings from the learnable normality and abnormality text prompts, capturing the underlying generic anomaly patterns.
3.3 Multi-Vies Rendering
Multi-view projection is crucial technology for understanding point clouds from perspectives. Some multi-view projection approaches project point clouds into various depth maps, providing adequate shape information for class recognition [59]. However, in this paper, our objective to learn both generic global and local anomaly semantics. Depth-map projection lacks sufficient resolution to represent fine- ane-grained anomaly semantics accurately. Hence, we adopt high-precision rendering to preserve the original 3D information offline. Specifically, given an auxiliary dataset of point clouds Daa (x?"d 3d ) we define the rendering matrix as (()) for the k-th view, with a total of K views. We simultaneously render point clouds and point-level ground truths from different views to obtain their corresponding 2D renderings, which is given by x! = (x:"
and y (k) (1²(d) where x RR x and (k) R H respectively represent the k-th renderings and corresponding pixel-level ground truth in the i-th point cloud. Note that anomaly
pixels are marked as I, and normal pixels are marked as
3.4 Representations for 3D and 2D information
PointAD aims to learn generic anomaly semantics from both 3D and 2D representations, enabling comprehensive understanding of point and pixel anomaly patterns. For a point cloud a:dd, (k) k Then, these renderings are encoded via the vision first obtain the 2D renderings X; = {at
encoder of CLIP obtain global 2D representations (f. }k=1- and local 2D representations |
FT = ( 1k=1- As for point cloud representations, we consider that one point cloud will be projected into multiple 2D renderings. Consequently, global 3D representation pt and local 3D representations p."" are expected to include their corresponding 2D representations in each (a) lp. (8) = (k)) jk=1 and p." {p." (M) m)(()) {p.. m=k 1K } where view. Formally, P. {p.
Pi (p=]() pm((h) (u, v) (*())) |b.g c,,)) represents the j-th point representation of i-th point cloud in the k-th view, whose 3D coordinate is (at.j, dij. G1j). (M))), is the rendering transformation between the point and pixel representation, derived as (k) R(()
m(()

---

Points at different positions may yield different number of 2D representations as they are hidden by other points from specific viewpoint. In this case, we introduce view-wise visibility mask M, where M indicates whether the j-th point of the i-th point cloud is visible in the k-th view. We compare the depth of points projected into the same pixel in the same view and set the corresponding visibility mask to for the point with the minimum depth, and to 0 for the other points. Let Q denote the depths set of all points that are projected into the same pixel indexed by (u,v) v) in the i-th point cloud in the k-th view. 2 and M, are respectively given as 2 = {or, I (u, 0)1 and M = j(i. a. (cu, ci, E 2, where I(.) is an indicator function. Local 3D representations of the i-th point cloud for the k-th view are reformulated as: P; m(k] mikk)
R (ac. ba.j.
(Pi) * M f==
3.5 Hybrid representation learning
The key of ZS 3D anomaly detection requires the model to capture generic anomaly semantics, rather than relying on specific object semantics. Since CLIP was originally pre-trained to align object semantics, such alignment harms the generalization capacity of CLIP to recognize anomalies on various objects. To adapt CLIP 3D anomaly detection, we propose hybrid representation learning, from both 3D and 2D perspectives, to globally and locally optimize textual representations. This enables PointAD to learn more representative text embedding for glocal anomaly semantics alignment. Following previous work 63 61], we randomly initialize two learnable text templates t, and ta in AnomalyCLiP [63] or CoOp manner [6]) to obtain more overall text embeddings Ir. and Ia recognize normality and abnormality, respectively.
b, [Vi] [Vr]] [object)
In [Vi] ... [VE][clas]]
[Wi] ... [We] [e]][dae [e]]daaged[ooojec))
L. [W1] [Wa Ww]]ddamaged[clas
PointAD
PointAD-CoOp
where V and W are learnable word embeddings, respectively.
MIL-based 3D representation learning To fully incorporate 3D glocal anomaly semantics into PointAD from point information. respectively devised two losses to capture 3D global anomalies and local anomaly regionals. First, we compute the cosine similarity between the textual represen- tation and its rendering global representations in each view. As point clouds are projected from different views, the resulting renderings in each view reflect certain parts of point clouds. We use view-wise MIL to integrate 2D global representations and then align global labels to capture the global semantics. Formally, the global 3D loss is defined as:
L global
N Σ; CrossEntropy K ES P(ge, ((K)), max )).")).
ad
As for local point anomaly semantics, we quantify the cosine similarity between textual representa- tions and local representations of 2D renderings. Since points within point clouds are projected from different views, their projections in each view present part characteristies of themselves. We adopt the pixel-wise MIL to achieve the aggregation of point local representation. The point segmentation can be formulated mathematically as follows:
((a) 1 Σ Seg(g.. ((())) std
k E. Seg(gn. m((k))
However, deriving such 3D segmentation requires similarity computation for each point. It brings significant memory burden, with a huge computational complexity of O O(Knd), which unaffordable for one NVIDIA RTX 3090 24GB GPU. To address this computational challenge, we resort to the rendering correspondence between points (3D space) and their corresponding pixels within each view
(2D space). We first can rewrite 3D segmentation from the view perspective as i(a) * E. sad(k) Then, the k-th division of 3D segmentation can be transformed into the 2D counterpart through the rendering projection sdd(k) = (R()))(- (ad(A) & ((()) where @ is the Hamiltonian product. The
i(a)
k-th 2D counterpart can be computed ((a) Up( Seg(ga: f()(())), where the operator Up(-) represents bilinear interpolation from feature space to 2D space. Finally, we can reformulate the 3D segmentation as follows:
- (17())) sm (m)()))) 8 M.")
sid = E.
From the equation, we can observe that the primary computation can be conducted in the feature space, with a computational complexity of O (Khwdd). This is a substantial overhead reduction compared to O(K (n) since feature space is much smaller than 3D space. i.e., h x w & n. In our experiment, h x w 24 x 24 576, while 336 x 336 112896. With this transformation, the
5

---

Table 1: Performance comparison ZS 3D anomaly detection in "one-vs-rest" setting.
Table 2: Performance comparison ZS M3D
anomaly detection in "one-vs-rest" setting.
Dotec:
Thraser
MVT==ID-ADX(0)
Fyecandies10)
ReaHD--AD(1))
Detee.
Datiset
MVTo-ID-ADX10)
Fyocandies(10
level
Menic
LAUROL
AF
I-AURIX."
AP
L'URRXX
AP
level
Mitric
FAUROC
AP
LAUROL""
AP
CR.I
61.2
85.8
66.7
69.2
68.8
72.3
CLIP + R.
60.4
86.4
73.0
73.9
Cheraghian
Cheraghian
PoinCLIPP V2
51.2
80.1
46.1
48.
58.
PomCLiP V2
49.8
79.3
46.9
49.9
6.
PoinCCTPP V2g
51.1
80.6
44.4
47.0
57.5
58.3
MG. PoinCC V2
49.4
79.8
48.5
50.5
AnomalyCLIP
56.4
83.5
$7.6
59,0
55.2
57.1
AnumalyCLIP
66.2
87.6
65.0
67.5
PuinAADCaaOp
PrepAA-CoOOp
PounAD
$2.0
94.2
69.1
73.8
74,8
76.9
PoinAD
86.9
96.1
77.7
30.4
Metric
P-AUROC
AUPRO
P-AUROC: AUPRO P-AUROC: AUPRO
Metric
PAAUROC AUPRO P-AUROC AUPRO
<
34,4
81.2
37.9
45.9
CLIP +
50.0
78.0
31,8
Cheraghian
88.2
57.0
Cheraghian
PoinCLIP
49.4
PoinCCLIP V2a.
87.3
52.3
44.2
52:2
ML. PounCCIP V2a
79.5
51.6
46.2
AnomalyCLIP
$8.9
60.9
72.7
43.4
50.3
AnemalyCLIPP
91.6
70.9
85.0
56.2
PolmAD-CaOp
PoozAD-CoOp
PoumAD
PointA
entire experiment can be conducted using only single NVIDIA RTX 3090 24GB GPU. After that, Dice Loss is employed to precisely model the decision boundary anomaly regions. Let represent a full-one matrix of the same size as v² Formally, we define 3D local loss L loool
Lidd
Li
Loral
R Σ; (Dice(S =²²') + (ic)(S)())> ")"
MTL-based 2D representation learning We further improve PointAD point understanding by capturing 2D glocal anomaly semantics into the object-agnostic text prompt template. We treat the anomaly recognition for one rendering from the point cloud as a task. Hence, we formulate the anomaly semantics learning for multiple 21) renderings as MTL. MTL.-based 2D representation learning is divided into two parts for respective alignment to 2D global and local anomaly semantics. For 2D global semantics, we use CrossEntropy to quantify the discrepancy between the textual representations and each global 2D representation. Global MTL-based 2D representation learning I.global defined as:
gloobal
Σi, CrosSEntrosy(P(g...1 ((*) max (v, ))
Also, we focus on 2D abnormal regions to understand pixel-level anomalies. As the anomaly regions are typically smaller than normal regions, we employ Focal Loss to mitigate the class imbalance besides Dice Loss. Let @ denote the concatenation operation. Local MTL-based 2D representation learningLLot is given as follows:
L local
NK E,, ( (200)) @ ((a) :9, () Dice(S ,I ) + Dice (((()) (k) .3:
3.6 Training and Inference
PointAD detects 3D anomalies from both 3D and 2D perspectives and thus combing these above losses to derive hybrid loss Lhybrid- We minimize Ligbrid to incorporate generic anomaly semantics into the text prompt from point and pixel spaces:
Tinorid = I.global
+ I.gd local
+ T.global + T
loca!
During training, we minimize the hybrid loss Laybrid, where the original parameters of CLIP are frozen to maintain its strong generalization. Since model provides unified framework to understand anomaly semantics from point and pixel, it can not only perform ZS 3D anomaly defection but also M3D anomaly detection in plug-and-play way. Next, we will introduce the inference process in detail:
ZS 3D/M3D inference Given a point cloud v.ddd we regard the 3D segmentation (See Equ. 2 the anomaly score map: =G, (S (S)))) where Go represents the Gaussian filter. The global anomaly
score incorporates glocal anomaly semantics and is computed A& { E P(g.,
max )). When the RGB counterpart is available for testing, PointAD could directly integrate RGB information by feeding RGB images to 2D branch to derive 2D representations. We project these 2D representations back to 3D branch to respectively compute RGB anomaly score map and
anomaly score as A. (rgbb) P(ge. f! ((rbb) and A, Ag(rgb) Go (S (d) The final multimodal
anomaly score map and anomaly score are defined as A, (modd) = (AT + A Am(rgh) and A. e(mod) = |l (AA + A,) + max (A ml(mod))) respectively.

---

Experiment
4.1 Experiment Setup
Dataset We evaluate the performance oS 3D anomaly detection on three public datasets including, MVTec3D-AD, Eyecandies and Real3D-AD. MVTec3D-AD, Eyecandies, and Real3D-AD multi- class datasets and respectively contain 10 classes, classes, and 12 classes. Since these training datasets only contain all normal samples, an object test dataset is used to fine-tune PointAD and assess the ZS anomaly detection for the remaining objects. We also explore more challenging setting: cross-dataset ZS generalization, which requires the detection model to generalize to anomalies on other datasets. For point cloud anomaly detection, we only use point clouds to detect and localize 3D anomalies. In M3D anomaly detection, the 2D RGB information is utilized only for testing. To comprehensively analyze PointAD, utilize four metrics assess its performance in both anomaly detection and segmentation.
the common zero-shot setting one-vs-rest, where
Input
View
View 2
View 4 score map score
View
View 5
View 6
View 7
View 8
View 9 score map
Point
Ground truth
score map
score map
score map
Visualization on anomaly score maps in ZS 3D anomaly detection. Point clouds of diverse objects are input into PointAD to generate 2D and 3D representations. Each row visualizes the anomaly score maps of 2D renderings from different views, and the final point score maps are also presented. More visualizations are provided in Appendix !
Figure 3:
4.2 Implementation Details & Baselines
Both point clouds and 2D renderings are resized to 336 x 336. We use Open3d library to generate 9 views by rotating point clouds along the X-axis at angles of - 0, fT, n & for most categories. We circularly set the rendering {- - - n. - - angles, evenly distributing the angles between CLIP model (VIT-L/11) 1-11L/146x6px in open_clipp Following [63], we improve the local visual se- mantics of vision encoder of CLIP without modifying its parameters. During training, parameters of CLIP frozen and set the learnable word embeddings in object-agnostic text templates to single NVIDIA RTX 3090 24GB GPU using PyTorch-2.0.0
x. The backbone of PointAD is the pre-trained
keep all
12. All experiments were conducted As there is no work to explore the field of ZS 3D anomaly detection, we make great effort to provide these comparisons. We apply the original CLIP to framework for 3D detection, called CLIP + Rendering. Also, we reproduce SOTA 3D) recognition works including PointCLIP V2
( and
Cheraghian [12), and adapt them for ZS 3D anomaly detection. We compare the SOTA 2D anomaly
detection approach AnomalyCLIP (63] by fine-tuning it on depth maps. PointAD by default uses object-agnostic text prompts, whereas PointAD-CoOp employs object-aware prompts. Appendix B and C provide more details on implementation and baselines.
4.3 Main Results
We fine-tuned PointAD on three objects on MVTec3ID-AD, Eyecandies, and Real3D-AD. Over three runs, the averaged results on one-vs-rest and cross-dataset settings are reported. We use the metric pairs (I-AUROC% % and AP% % 1) and (P-AUROC% % and AUPRO% % 1) to evaluate the glocal detection performance, respectively. Details experimental settings see Appendix [A] The best and second-best results in ZS are highlighted in Red and Blue. G. and L. represent 3D global and local anomaly detection. M3D global and local anomaly detection are abbreviated as MG. and ML.
ZS 3D anomaly detection Table [ presents the comparison of ZS 3D performance. Compared to the point- method Cheraghian and the projection- based method PointCLIP V2, PointAD achieves superior performance on ZS 3D anomaly detection over all three datasets. Especially, outperforms CLIP + Rendering from 61.2% to 82.0% I-AUROC and from 85.8% to 94.2% AP on MVTec3D-AD. In addition, PointAD achieves superior segmentation performance on 7S 3D anomaly detection, improving MVTec3D-AD by large margin compared to Cheraghian from 88.2%
7

---

RGB
Ground truth
Point
Muttimodal score map
20
Point cloud Input
Ground truth
RGB
Point
Muttimodal score map
Input
Input
score map
score map
Input
score map
score map
(a) Multimodal visualization with hybrid loss.
(b) Multimodal visualization without 2D glocal loss. Figure 4: Visualization comparison between PointAD with hybrid loss and without.
to 95.5% P-AUROC and from 57.0% to 84.4% AUPRO. This improvement in overall performance is attributed to PointAD adapting CLIP's strong generalization glocal anomaly semantics through hybrid representation learning. In addition, PointAD advances PointAD-CoOf, across all datasets by blocking the class semantics in text prompts
7S M3D anomaly detection We also compare the ZS M3D anomaly detection when RGB infor- mation is available for testing. As shown in Table 2 the results indicate that PointAD can integrate additional RGB information and further boost its performance from 82.0% to 86.9% AUROC and from 94.2% to 96. 1% AP for global semantics on MVTec3D-AD. Additionally, as for local semantics, the performance improves from 95.5% to 97. 97.2% P-AUROC and from 84.4% to 90.2% AUPRO. A large performance gain is also obtained on Eyecandies and Real3D-AD. While other methods improve their performance in some metrics, they still suffer from performance degradation in other metrics due to inefficient integration of the two modalities. Instead, PointAD achieves overall improvement across all metrics by incorporating explicit joint constraints on both point and pixel information.
Cross-dataset 7S anomaly detection We perform the cross-dataset anomaly recognition to further eval- uate the zero-shot capacity of PointAD, where use one object as the auxiliary and test objects with totally different semantics and
Table 3: Performance compari- son on ZS 3D anomaly detection in cross-dataset setting.
Table 4: Performance compari- son on ZS M3D anomaly detec- tion in cross dross-dataset setting.
Detec.
Dalaset
Eyecandies(1))
(111)
Detec. level
Duiaset
Eyecandies(10)
level
Metric
FAURXX
AP
LAUROC
AP
Matnic
EAUROC
AP
PoinCCIIP
45.2
480
57.4
58.8
PainCCIJP V2
50.9
AnomalyCLIP
56.3
57.1
52.7
55.7
AnomalyCLIP
65.7
68.1
PoinAD-CoOp
69.1
738
748
769
ProtAADCssOp
76.3
78.9
PoinAD
69.5
74.3
75.9
77.9
PointAD
78.6
80.8
Metric
P-AUROC AUPRO P-AUROC AUPRO
Matric
P-AUROC ALPRO
scenes in another dataset. We compare all baselines that need fine-tuning. From
LiP V2g
AnomalyCLiP
43.9
79.5
45.4
519
50.3
ML.
PounCLIP V2.
AnomalyCLIP
86.2
61.3
PoinAD CoOp
91.8
70.5
70.1
PeEmLAD PeonAD-CooOp
94.
80.3
PoinAD
71.4
71.6
PointAD
Table 3 and M3D) from
Table PointAD demonstrates strong cross-dataset generalization performance on Eyecandies and Real3D-AD, with nearly no obvious performance decay compared to the one-vs-rest setting. The strong transfer ability highlights its robust generalization capabilities in detecting anomalies in objects with unseen semantics and backgrounds.
4
4.4 Result Analysis
Visualization analysis. To intuitively present the strong generalization capacity of our model to unseen anomalies, we visualize the anomaly score maps of the 3D and corresponding 2D counterparts of PointAD MVTec3D-AA. As shown in Figure PointAD reveals abnormal spatial relationships of points and further captures the generic point anomaly patterns across diverse objects. And, we also visualize the anomaly score map of corresponding 2D counterparts, where 3D point anomalies transformed into 2D pixel anomalies. It can be observed that PointAD also has a strong detection ability on such 2D anomalies. The strong representative pixel representations from multiple views facilitate more precise 3D anomaly detection. Quantitative results are provided in Section 5 The strong 3D and 2D detection capabilities of PointAD from hybrid representation learning, which only enables PointAD to capture the 3D anomalies but also explicitly constrains 2D representations.
How multimodality makes PointAD accurate. PointAD unified framework that can not only capture point anomalies but also handle 2D information in plug-and and-play manner. As shown in Figure (a) we visualize M3D results of PointAD on --Te The surface damage on the potato presents similar appearance to the object foreground, which makes it difficult to detect this anomaly with RGB information. On the contrary, the point relations for the color stain on foam are the same those of normal, but they have clear distinction in the RGB information. PointAD
8

---

can integrate these two modalities, thereby complementing their respective advantages. We further investigate the reason why PointAD can directly leverage both modalities. For this purpose, we experiment without 2D glocal loss, As shown in Figure[4(b) without 2D glocal loss, significant noise disrupts and even covers the RGB score maps, resulting in unpromising multimodal fusions. This illustrates the importance of explicit constraints on the 2D space. Hence, we conclude that the robust multimodal detection capability of our model stems from the collaboration optimization in both 3D and 2D spaces during training. We provide more analysis about failure cases and the computation overhead in Appendix G and
5 Ablation Study
Module Ablation. Here, we investigate the effectiveness of the proposed main technologies by progressively adding the proposed modules. Table 6 iillsstraees that vanilla. which represents the aforementioned CLIP + rendering, performs poor results on both 3D and M3D anomaly detection because CLIP focuses on alignment for 2D object semantics instead of anomaly semantics. With the 3D global branch, we incorporate the global anomaly semantics into PointAD, improving overall performance in local and global detection. After adding the 3D local branch, the performance is further improved, while the pixel-level performance on M3D detection suffers from performance degradation. This is attributed to the absence of 2D constraints, leading to inefficient multimodality fusion as we integrate 2D RGB information in plug-and-play way. The inclusion of 2D global branch explicitly incorporates 2D anomaly information, which makes PointAD obtain overall performance gain. Finally, by further focusing on 2D anomaly regions, PointAD has a deeper understanding of point clouds from 2D representations and promotes multimodality fusion. Therefore, our model notably boosts the multimodal segmentation performance from 92.9% to 97. 7722% P-AUROC and from 84.4% to 90.2% AUPRO.
Table 6: Ablation on the proposed modules.
Table 5: Ablation on rendering number.
Module
Point detection
Mulimodal detection
View number
Local
Point desection
Global
Multimodal detection
Local
Global
Ligh L.
L.ghhhak Locul Laohbal Loput
Local
Global
Local
Global
(9.11.9.1)
(72.6 90.1)
(96.0 87.6)
(80.4, 93.9)
(. 54.4)
(61.2, 85.8)
(. 56D))
(60.4, 86.4)
(9..2,82.5
(76.8 92.1)
(96.9. 89.5)
(83.7, 95.1)
(75.5.91.)) (9.9.71.77
(92 81.6)
(80.4.933
(95.3, 84.3)
(808, 93.8)
(97.1. 89.5)
(85.9.95.7
38.33
81.4)
(83.9. 95.0)
93.9)
(97.3,
(86.5, 95.9)
(93.9, 82.8)
(79.5,91.6)
(910) 82.2)
(82.6, 94.1)
(95.5, 84.4)
(82.0. 96.2)
(91.2. 90.21
(86.9.96.1)
(95.5, 84.7)
(81.8,92.3 92.3) (96 90.6)
(83.7, 95.1)
1I
(9..4,838)
(81.7, 91.2)
(97.1, 90.1)
(85.4,95.)
(95.6 82.5)
(82.499.5) 94.5)
(92.9 84.4)
(85.5, 95.6)
(95.5
(82.0, 91.2)
(97.2, 90.2)
(86.9, 96.1)
View Number Ablation. PointAD interprets point clouds from 2D renderings, and the quantity of rendering views directly affects the 3D original information acquired by PointAD. Tadpetssdepicts that the appropriate number of views benefits point understanding from informative views while alleviating negative effects of subpar views. More ablations about the length of learnable prompts, layers of intermediate vision features, and the number of training sets are provided in Appendix F
6 Conclusion
This paper takes the first attempt to study the challenging yet underexplored tasks of ZS 3D and M3D anomaly detection. We propose a unified framework, namely PointAD, to transfer the strong generalization of CLIP to 3D point clouds. PointAD understands 3D anomalies from 3D and 2D spaces. Benefiting from hybrid representation learning. PointAD can recognize generic 3D normality and abnormality across diverse objects and directly integrate RGB information for ZS M3D. Extensive experiments demonstrate the superior ZS detection capacity of our model, whether single modality or multimodality. Code will be made available once the paper accepted.
Limitations PointAD utilizes fixed rendering angles to generate 2D renderings across diverse objects. While experimental results demonstrate its superiority, the development of a fine-grat fine-grained filtering mechanism to select high-quality 2D renderings, particularly for revealing anomalies, remains an avenue for future research.
Broader Impact Our paper aims to enhance automated detection and decision- decision-making in smart manufacturing, which does not involve any potential ethical risks. Since the collection of 3D samples is more labor-intensive and costly, our research on using vision-language models for zero-shot point cloud detection can have significant societal impacts, especially in scenarios where target 3D training samples are unavailable due to privacy concerns or the absence of products. We hope that our first exploration of the ZS 3D anomaly detection field could pave the way for further research this emerging field.

---

Acknowledgments
This work was supported by NSFC U23A20326 and NSFC 62088101 Autonomous Intelligent Unmanned Systems.
References
[1] Toshimichi Aota, Lloyd Teh Tzer Tong, and Takayuki Okatani. Zero-shot versus many Unsupervised texture anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5564 55744-557,440 2023.
[2] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9592-9600) 2019.
[3] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4183 4192, 2020.
[4] Paul Bergmann and David Sattlegger. Anomaly detection in 3d point clouds using deep geometric descriptors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2613-2623, 2023.
[5] Luca Bonfiglioli, Marco Toschi, Davide Silvestri, Nicola Fioraio, and Daniele De Gregorio. The eyecandies dataset for unsupervised multimodal anomaly detection and localization. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 3586-3602, December 2022.
[6] Tri Cao, Jiawen Zhu, and Guansong Pang. Anomaly detection under distribution shift. arXiv preprint arXiv:2303. 13845, 2023.
[7] Yunkang Cao, Xiaohao Xu, and Weiming Shen. Complementary pseudo multimodal feature for point cloud anomaly detection. arXiv preprint arXiv:2303.13194, 2023.
[8] Yunkang Cao, Xiaohao Xu, Jiangning Zhang, Yuqi Cheng, Xiaonan Huang, Guansong Pang, and Weiming Shen. A survey visual anomaly detection: Challenge, approach, and prospect. ArXiv, abs/2401.16402, 2024.
[9] Ruitao Chen, Guoyang Xie, Jiaqi Liu, Jinbao Wang, Ziqi Luo, Jinfan Wang, and Feng Zheng Easynet: An easy network for 3d industrial anomaly detection. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7038-7046, 2023.
[10] Xuhai Chen, Yue Han, and Jiangning Zhang. A zero-/few-shot anomaly classification and segmentation method for evpr 2023 vand workshop challenge tracks 1&2: 1st place zero-shot ad and 4th place on few-shot ad. arXiv preprint arXiv:2305.11388 2023.
[1I] Yuanhong Chen, Yu Tian, Guansong Pang, and Gustavo Carneiro. Deep one-class classification via interpolated gaussian descriptor. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 383-392, 2022.
[12] Ali Cheraghian, Shafin Rahman, Townim Chowdhury, Dylan Campbell, and Lars Petersson. Zero-shot learning on 3d point cloud objects and beyond. International Journal of Computer Vision, 130(10):2364 2384, 2022.
[13] Yu-Min Chu, Chich Liu, Ting-I Hsieh, Hwann-Tzong Chen, and Tyng-Luh Liu. Shape-guided dual-memory learning for 3d anomaly detection. In International Conference Machine Learning, pages 6185-6194. PMLR, 2023.
[14] Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspon- dences. arXiv preprint arXiv:2005.02357. 2020.
[15] Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embed- ding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9737-9746, 2022.
10

---

[16] Hanqiu Deng, Zhaoxiang Zhang, Jinan Bao, and Xingyu Li. Anovl: models for unified zero-shot anomaly localization. arXiv preprint arXiv:2308. 2023.
Adapting vision- language
[17] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. sero oft-of-distribution detection based on the pre- tre-trained model clip. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 6568-6576 2022.
[18] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In International Conference on Machine Learning, pages 3809-3820. PMLR, 2021.
[19] Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anoma- lygpt: Detecting industrial anomalies using large vision-language models, 2023.
[20] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 98-107, 2022.
[21] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1-11, 2021.
[22] Eliahu Horwitz and Yedid Hoshen. Back to the feature: classical 3d features are (almost) all you need for 3d anomaly detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2967-2976, 2023.
[23] Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang. Registration based few-shot anomaly detection. In European Conference on Computer Vision, pages 303-319. Springer, 2022.
[24] Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19606-19616. 2023.
[25] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint 2rXiv:2304.026633 2023.
[27] Aodong Li. Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Zero-shot anomaly detection via batch normalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[28] Wenqiao Li, Xiaohao Xu, Yao Gu, Bozhong Zheng, Shenghua Gao, and Yingna Wu. Towards scalable 3d anomaly detection and localization: A benchmark via 3d anomaly synthesis and self-supervised learning network. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22207-22216 2023.
[29] Xurui Li, Ziming Huang, Feng Xue, and Yu Zhou. Muse: Zero-shot industrial anomaly classifi- cation and segmentation with mutual scoring of the unlabeled images. ArXiv, abs/2401.16753. 2024.
[30] Yiting Li, Adam David Goodge, Fayao Liu, and Chuan-Sheng Foo. Promptad: Zero-shot anomaly detection using text prompts. 2024 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1082-1091, 2024.
[31] Jiaqi Liu, Guoyang Xie, ruitao chen, Xinpeng Li, Jinbao Wang, Yong Liu, Chengjie Wang, and Feng Zheng. Real3d-AD: dataset of point cloud anomaly detection. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
11

---

[32] Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller. Explainable deep one-class classification. arXiv preprint arXiv:2007.01660, 2020.
[33] Philipp Liznerski, Lukas Ruff, Robert Vandermeulen, Billy Joe Franks, Robbrt Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint arXiv:2205. 1474, 2022.
[34] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: prrXiv:331..30 1301.3881, 2013.
[35] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. Advances in Neural Information Processing Systems, 35:35087-35102, 2022.
[36] Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van den Hengel. Explainable deep few-shot anomaly detection with deviation networks. arXiv preprint arXiv: (0800046,, 2021.
[37] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1-38, 2021.
[38] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[39] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8888 8763. PMLR, 2021.
[41] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guide dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082-18091, 2022.
[42] Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2806-2814, 2021.
[43] Tal Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2155-2162, 2023.
[44] Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14318-14328, 2022.
[45] Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Asymmetric student- teacher networks for industrial anomaly detection. In Proceedings of the IEEF/CVE Winter Conference Applications of Computer Vision, pages 2592-2602, 2023.
[46] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pages 4393-4402, PMLR, 2018
[47] Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John Shawe- Taylerr John C Platt, et al. Support vector method for novelty detection. In NIPS, volume 12, pages 582-588. Citeseer, 1999.
12

---

[48] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Searned-Milles Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945-953,, 2015.
[49] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations, Advances in Neural Information Processing Systems, 35:30569-30588 2022.
[50] Yiyou Sun, Zhenmei Shi, Yingyu Liang, and Yixuan Li. When and how does known class help discover unknown ones? provable understandings through spectral analysis. In International Conference on Machine Learning, 2023.
[51] Yu Tian, Fengbei Liu, Guansong Pang, Yuanhong Chen, Yuyuan Liu, Johan W Verjans, Ra- jvinder Singh, and Gustavo Carneiro. Self-supervised pseudo multi-class pre-training for unsupervised anomaly detection and segmentation medical images. Medical Image Analysis, page 102930, 2023.
[52] Yu Tian, Guansong Pang. Fengbei Liu, Yuanhong Chen, Scon Ho Shin, Johan W Verjans, Rajvinder Singh, and Gustavo Carneiro. Constrained contrastive distribution learning for unsu- pervised anomaly detection and localisation in medical images. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part V 24, pages 128- 140. Springer, 2021.
[53] Chengjie Wang, Haokun Zhu, Jinlong Peng, Yue Wang, Ran Yi, Yunsheng Wu, Lizhuang Ma, and Jiangning Zhang. M3dm-nr: Rgb-3d noisy-resistant industrial anomaly detection via multimodal denoising. arXiv preprint arXiv:2406.022 02263, 2024.
[54] Yue Wang, Jinlong Peng, Jiangning Zhang, Ran Yi, Yabiao Wang, and Chengjie Wang. Mul- timodal industrial anomaly detection via hybrid fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[55] Guoyang Xie, Jingbao Wang, Jiaqi Liu, Feng Zheng, and Yaochu Jin. Pushing the limits of fewshot anomaly detection in industry vision: Graphcore arXiv preprint arXiv:2301. 12082, 2023.
[56) Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-elass anomaly detection. Advances in Neural Information Processing Systems, 35:4571-4584, 2022.
[57] Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows. arXiv preprint arXiv:2111.07677, 2021.
[58] Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Cheating depth: Enhancing 3d surface anomaly detection via depth simulation. 2024 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 2153-2161, 2023.
[59] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for dision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for langunelanguage models. International Journal of Computer Vision, 130(9))2337-2348, 2022.
[62] Qihang Zhou, Shibo He, Haoyu Liu, Tao Chen, and Jiming Chen. Pull & push: Leveraging differential knowledge distillation for efficient unsupervised anomaly detection and localization. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
13

---

[63] Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object- agnostic prompt learning for zero-shot anomaly detection, 2023.
[64] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointelip v2: Prompting clip and gpt for powerful 3d world learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2023.
14

---

Rendering angles A Dataset
Dataset We evaluate the performance of ZS 3D anomaly detection on three publicly available 3D anomaly detection datasets, MVTec3D-AD, Eyecandies, and Real3D-AD. MVTec3D-AD comprises 4147 point clouds across 10 categories. These objects exhibit diverse object semantics, including bagel, cable gland, carrot, cookie, dowel, foam, peach, potato, rope, and tire. The training dataset comprises 2656 normal point clouds, and the validation dataset comprises 294 normal point clouds. The test dataset includes 948 normal and 249 anomaly point clouds, covering several anomaly types. Point-wise annotations are available for the point clouds. MVTec3D-AD also provides corresponding 2D-RGB image counterparts for the point clouds. We remove the background plane of point clouds in the whole dataset like [22]. Eyecandies also has different classes and provides the corresponding 2D RGB information. Real3D-AD is recently available dataset, which contains 12 objects. However, does not provide the RGB information.
Evaluation Setting and Metric Since the training dataset of MVTec3D-AD only contains all normal samples, we use an object test dataset as the auxiliary dataset to fine-tune PointAD and assess the ZS anomaly detection for the remaining objects. In particular, we report the average results using different objects as the auxiliary, i.c., carrot, cookie, and dowel for MVTec3D- AD; confetto, LindriceSandwich, and PeppermintCandy for Eyecandie; seahorse, shell, and starfish for Real3D-AD. Moreover, we explore more challenging cross-dataset generalization settings, where we use auxiliary data to test all objects of another dataset. For point cloud anomaly detection, we only use point clouds to detect and localize 3D anomalies in Figure ((a) In M3D anomaly detection, both point clouds and their 2D-RGB counterparts are utilized, as shown in Figure 5(b) To comprehensively analyze PointAD, we utilize four metrics to assess its anomaly classification and segmentation performance. For anomaly detection, we use the Area Under the Receiver Operating Characteristic Curve (I-AUROC% 1) and average precision (AP% 1). Regarding anomaly segmentation, we use point-level AUROC (P-AURCC% 1) and a restricted metric called to provide detailed evaluation of subtle anomaly regions.
AUPRO%(1) B
Zero-shot 3D anomaly detection
Zero shot multimodal anomaly detection
prompts
R68 counterport
prompts
Point clouds
Point clouds
R-1 -
Anomaly score
Anomaly score
2D branch
branch
branch
Anomaly map
Anomaly map
(a) Inference for ZS 3D anomaly detection
(b) Inference for ZS M3D anomaly detection
Figure Inference schematic for ZS 3D and M3D anomaly detection.
B
Implementation Details
Table 7: Ablation study on the number of rendering views.
View number
f O,
5 0, S 3
n.
1,
17,
0,
m 0,
Both
point clouds and 2D renderings are resized 336 x 336. We Open3d li- brary to generate views by rotating point clouds along the X-axis at angles of
httpss//ggihuu..om///slooo//////
15

---

(- - 0, | 6" 5 for most categories. Some categories lose their surface completely because they are not stereo point clouds. Table also gives the specific rendering an- gles of different view number settings. We set the rendering angles in circular manner, evenly distributing the angles hetween π to æ. The backbone of PointAD the pre-trained CLIP mode F
(VIT-L/143336px) Following [63]. we improve the local visual semantics of the vision encoder of CLIP without modifying its parameters and introduce learnable tokens the text encoder. During training, we keep all CLIP parameters frozen and set the learnable word embeddings in object- agnostic text templates to 12. We use the Adam optimizer with learning rate 0.001 to optimize the learnable parameters. The experiment runs for 15 epochs with batch size of 4. All experiments were conducted on a single NVIDIA RTX 3090 24GB GPU using PyTorch-2.0.0.
C Baselines
ZS 3D anomaly detection and M3D) anomaly detection have not yet been explored. We first make an adaption for the original CLIP for 3D anomaly detection. Then, we reproduce SOTA ZS 3D classification methods (i.e., PointCLIP V2 [64 and Cheraghian (12]) and adapt them our settings. SOTA unsupervised 3D anomaly detection approaches are reported as the performance upper bound. All hyperparameters in these baselines are kept the same. We will present the detailed reproduction as follows:
CLIP + Rendering is a method, where we apply the original CLIP into our framework for 7S 3D anomaly detection. It uses the same rendering procedure as PointAD, Follow- ing [24, 63), we integrate anomaly semantics into CLIP by two class text prompt templates: A photo of normal [cls] and A photo of anomalous [cls], where cls de- notes the target class name.
PoinCLIP V2 (CVPR 2023) is SOTA ZS 3D classification method based CLIP, they project point clouds into depth maps from different views. To adapt PointCLIP V2 into replace its original text prompts point cloud of big [c]
ZS anomaly detection, with normal text prompts point cloud of a big [c] and abnormal text prompts point cloud of big damaged [c].
AnomalyCLIP (ICLR 2024) is a SOTA zero-shot 2D anomaly detection method. Anomaly- CLIP introduces object-agnostic learning to capture generic anomaly semantics of We adapt AnomalyCLIP in 3D detection by fine-tuning AnomalyCLIP on depth images of corresponding point clouds.
images,
Cheraghian (IJVV 2022) is an approach for ZS 3D classification without foundation models. They directly extract the point presentations by PointNet and use word2vector [34] to generate the textual embedding of an object. To incorporate the anomaly semantics into Cheraghian, we average the textual embeddings of [c] and damaged. We replace the global representation with dense representations to provide the segmentation to provide the segmentation results.
D Related Work
D.1 2D Anomaly Detection
2D anomaly detection has been studied extensively by leveraging RGB information 47
[52 [1 62] 43). Related works can be categorized into two branches: end-to-end and memory- based methods. Representative end-to-end methods exploit knowledge distillation [ 62
45 and
normalizing flow [20 57) to model the normal distribution. Instead, memory-based methods
55
store normal features to construct normal prototypes. 7S 2D anomaly detection proposed to target a challenging problem where training samples are inaccesible [17 33
16 10]. WinCLIP 24
attempts to explore ZS 2D anomaly detection using CLIP. AnomalyCLIP 63 first introduces object-
agnostic prompt learning to capture generic normality and abnormality, detecting anomalies across
datasets. PromptAD 30 focuses on effectively fusing these embeddings to enhance zero-shot
detection performance.
16

---

Original
Sigma
Sigma
Sigma
20%
30%
50%
70%
Original
Ground Truth
Ground
Viusalization with different rendering quality. larger o represents poorer rendering quality.
Figure 6:
Figure 7: Visualization with different resolutions. We downsample entire point clouds with different ratios to obtain diverse resolutions.
Table 8: Analysis on the rendering quality. The original setting highlighted in gray.
Table 9: Analysis on the input solution.
Dowssampl:
Print detectioon
Multimadal detecrion
Blur
Point detection
Mulkimodal detecsion
ratio
Global
Local
Global
Local
sigma
(920, 2)
Global
(95.5, 54.4)
(R05) 96.13
00.2, 90.23
Local
70%
(82.0,
(80.5 93.3) (95 82.6)
84.4)
(86.2. 95.1)
077.2
(97.0. 90.0
(80.1,
(95.2 83.2)
(83 95.6)
(92.0 90.00
50%
(77.9
(95.0 82.0)
(83.6 94.9)
(9688 89.2
(78.2, 92.5)
(836,
(74.6,
(94.9, (1.44
(819.
(77669922)
(95.1, 82.3)
(83.2 95.0)
(95.5, 89.3)
20%
(73.9.909)
(94.3.7877)
(8033 93.4)
(25.3, 87.5)
E Analysis on Rendering Conditions
Rendering quality PointAD interprets point clouds through their corresponding 2D renderings, and the quality of these renderings impacts the information that PointAD can extract from the original point clouds. In our manuscript, we used the Open3D Library to render the point clouds, but it does not provide an API for controlling rendering quality. To simulate varying rendering quality, we applied Gaussian blur with different extents to the 2D renderings. Sample visualizations are included Figure 6 Specifically, we conducted experiments on MVTec31--AD using different blur values (i.e..) ). Table ( shows that the detection performance of PointAD diminishes as rendering quality decreases (with increasing sigma). However, the degradation is acceptable even when the renderings are heavily blurred (a equals 9). In such cases, PointAD still outperforms basclines that use high-quality renderings.
Input Resolution Here, study the effect of resolutions of input point clouds. To create low- resolution point clouds, we downsample the entire high-resolution point clouds using Farthest Point Sampling (FPS) with various sampling ratios. This strategy allows generate corresponding low-resolution datasets for training and evaluating PointAD. Visualizations of these datasets are provided in Figure 7 We train PointAD using the resulting low-resolution samples and test PointAD on the same resolution. Table demonstrates that PointAD maintains strong detection capacity for low-resolution point clouds when the downsampling ratios are 20%, 30%. 50%, and 70%. Even at 20% resolution, PointAD D still achieves state-of-the-ae performance. This indicates that PointAD is generally applicable point clouds with various resolutions.
Rendering angles and different numbers of views PointAD interprets point clouds from their 2D renderings, where rendering angles and numbers collectively determine the amount of information derived. They have different emphases. For rendering angles, the importance lies in the discrepancy between adjacent angles, this affects the information granularity that PointAD obtains from adjacent views. When the angle discrepancy is fixed, the number of renderings determines the coverage of information in the resulting 2D renderings. To capture point cloud information, especially abnormal points, it is crucial to ensure comprehensive coverage. Therefore, our approach in selecting rendering angles and the number of renderings is to guarantce that all points in point clouds are adequately represented.
Based on this principle, we conducted experiments assess the impact of the number of renderings on PointAD's detection performance, circularly rendering point clouds to ensure even coverage of all points. As shown in Table [5] increasing the number of views allows PointAD to gather more detailed information from the 2D renderings, benefiting from smaller angle discrepancies, which improves detection and localization results. However, when the number of views increased from to 11, we observed a performance decline in PointAD, with the I-AUROC for global multimodal detection dropping from 87.4% to 86.4%. This suggests that incorporating too many views could introduce redundant information, resulting in 2D renderings with extensive overlap and excessive
17

---

Table 12: Analysis on the point occlusions.
Point detection
Multimodal detection
Method
Global
Local
Global
Local
original
(82.0, 94.2)
(95.5, 84.4)
(86.9, 96.1) (97.2, 90.2)
occlusions
(73.3, 90.6)
(94.3, 80.8)
(83.0, 94.8)
(96.7, 89.5)
local detail. This overemphasis on local information can impede global recognition. Hence, the appropriate number of views benefits point understanding from informative views while mitigating the adverse effects of redundant local information. To further explore the impact of the absolute angle, we shift the rendering angles while keeping the angle discrepancy unchanged. The original adjacent angle discrepancy in our paper is π. We divide this discrepancy into four parts and perform angle shifts of 20 20 R. and 20 ग to test the impact of varying rendering angles. Table 10 shows that PointAD maintains consistent performance across different rendering angles, demonstrating its robustness to variations in angles different from those used during training.
2
Table 10: Analysis on the rendering angle.
Table 11: Analysis on the rendering lighting.
Angle
Point detection
Mulimodal detection
Lightin
detection
Multimodal detection
Global
Local
Global
Global
Local
Global
Local
[S2.0, 94.23
(55.5, 54.4)
(860, 96.13
(97 90.23
94.3)
(95.4 83.7)
(85.7. 98.8)
(973, 90.75
182.6
90.5)
(82.1, 94.4)
(95.4, 84.1)
(86.4, 95.9)
(97.1, 90.7)
original
182.) 94.2)
(82.4, 94.3)
(95.3, 83.9
195.5 54.09
(86.4 95.9)
(6.3996.1)
(97.2, 90.23
(9.1, 90.07
(82.6. 94.61
(95.4,
(97.1, 98694.959)
(86.1.95.
Lighting--
Lighting-
Original
Lighting *
Lighting+*
Original
Occlusion
Ground Truth
Ground Truth
Figure 9: Visualization of oc- cluded point clouds.
Figure 8: Visualization with different rendering lighting.
Rendering lighting Further exploring the robustness of PointAD under different conditions could enhance its generalized detection performance. We conducted ablation studies to test its sensitivity under different lighting conditions and with occluded point clouds below. To evaluate the impact of rendering lighting, we adjusted the lighting conditions to render point clouds, generating variant datasets with different lighting. We used both stronger and weaker lighting to render point clouds compared to the original dataset, covering broad lighting range. We denote stronger and the strongest lighting as "+" and "++", and weaker and the weakest lighting as "_" and "_". Visualizations of the resulting samples where we tested PointAD, trained on the original dataset, on these lighting variant datasets. Table shows that PointAD can still detect anomalies even with significant discrepancies in rendering lighting, suggesting that PointAD not sensitive to variations in rendering lighting.
presented in Figure The experiments were conducted MVTec3D-AD.
Point occlusions Next, we evaluated detection performance with occluded point clouds. We occluded point clouds by removing those invisible from a specific rendering angle and then used the same rendering parameters to project the remaining points. During this process, we observed that abnormal regions might be occluded totally. Unlike class classification, where class semantics remain unchanged by point occlusions, removing these anomaly semantics would transform anomaly samples into normal points. Therefore, we selected rendering angles that allow visibility of part or all anomalous regions when the point cloud an abnormal instance. The occluded point clouds are shown in Figure We then used PointAD, trained on the original dataset, to test the resulting occluded dataset. Table [2 shows that PointAD suffers from performance degradation when the point clouds occluded. We attribute this to two aspects: 1) Despite this strategy, the occluded
18

---

point clouds could still lose part of the anomaly semantics; 2) Occluded point clouds could create unexpected sinkholes on the surface, which may cause PointAD identify these areas as hole anomalies incorrectly.
F Hyperparameter Ablation
Table 13: Ablation study on the length of the learnable prompt.
Length of learnable prompt
Point detection
Multimodal detection
Pixel level
Image level
Pixel level
Image level
(94.6, 83.4)
(81.7, 94.2)
(96.5, 89.8)
(86.6, 96.0)
(95.2, 83.6)
(82.0, 94.2)
(96.8, 90.0)
(86.6, 95.8)
10
(95.3, 84.0)
(81.8, 94.3)
(97.0, 90.1)
(86.5, 96.0)
12
(95.5, 84.4)
(82.0, 94.2)
(97.2, 90.2)
(86.9, 96.
14
(95.4, 83.7)
(814, 94.1)
(96.9, 89.7)
(85.5, 95.6)
(95.1, 83.0)
(81.5, 94. I)
(96.9, 89.8)
(84.7, 95.6)
Length of learnable prompts We study the sensitivity of important hyperparameters in PointAD. First, we explore the length of learnable text prompt templates, as shown Table. [3] As the length of word embeddings increases, PointAD can better learn 3D and 2D anomaly semantics to improve its performance. Nonetheless, with a further increase in length (i.e., from 12 16), a decline in performance becomes noticeable. The excessive or insufficient number of learnable word embeddings can lead to performance degradation. An appropriate length (i.e., 12) is important for PointAD to attain comprehensive performance in both 3D and M3D anomaly detection.
Training set We have increased the test data for each category to incorporate more instances on MVTec3D-AD. Originally, we evaluated the rest of the data using only one category as the test data, such as carrot, cookie, and dowel. Now, we attempt to incorporate more instances. Here, we utilized two categories as auxiliary data, including carrot and cookies, carrot and dowel, and cookie and dowel. To further analyze the effect of the size of auxiliary data, we utilized three categories as auxiliary data, selecting all possible combinations from the four sets: carrot, cookie, bagel, and dowel. We present the performance averaged across all groups below. From Table 14 and Table 15] PointAD can incorporate more knowledge about abnormality and normality, improving point detection and multimodal detection performance. Specifically, from one category to three categories, PointAD exhibits improved performance, with P-AUROC increasing from 95.5%, 96.1%, to 96.3%, and AUPRO increasing from 84.4%, 86.3%, to 86.5%. Moreover, I-AUROC increases from 97.2%, 97.5%, to 97. 97.8%, and AP increases from 90.2%, 91.8%, to 92.0%. This trend is also observed in global anomaly semantics.
Table 14: Ablation study on training set size for point detection
Training sct
Bagel
Cable gland
Carrot
Cookie
Dowel
Feam
Peach
Potato
Rape
The
Mean
one
198.3, 99.61 (58.) 86.00 (9799 99.6) 1911, 97.99 (12.2, 91.00 (09.5, 91.2) (91.5, 97.6) (98.8, 99.7) (91.5, 96.7) (54.1, $2. (82.0. 94.2)
G.
two
199.0, 99.8 (52.5, 85.3) (98.1, 99.7) (9L8, 97.7) (70.2.91. (68.9 90.8) (91.5,97 (99 99.8) (908, 96.5) (56.9, 85.0) (81.9 91.4)
(100, (52.2, 84.8)
(94.2,
91.6)
91.1) (92.6,
(889) (54.5, 83.6)
198.4, 96.9) (93.5, 79.5) (99.4 96.4) (87.5, 75.4) (95.5, 75.2) (86.5, 56.1) (99.5, (99.9) (99.3, 89.9) (95.3, 79.7)
(95.5,
two
(99. 1, (90.5, 79.91 (99.6, 97.7) (90.4,
75.2) (88.3, 57.3)
(99.6, (99.9, 99.6) 91.00 (96.0,
(96.1,
three
(99.3 3, 98.2) (93.5, 79.89 (99.6, 97.8) (92 87.7) (95.7, 75.7) (87.9) 57.1) (99 98.7) (99.9, 99.0) (99.4, 90.6) (95.7, 80.3)
(96.3, 86.5)
Table 15: Ablation study on training set size for multimodal detection.
Training
Bugel
Cable gland
Carrot
Cookie
Dowel
Foam
Peuch
Potato
Tire
Mean
(9R 8, 99.7) (79.9. 94.7) (95.5, 98 (86 95 5) (98.5 90.6) (84.4, 96, (96 99, 1) (90.7, 97.0) (93.6, 97.3) (74.6, 92.0) (B6 96.1)
MG
R,
99.6)
(95
(94.5, 97. (77.2,
throe
(98.6, 99.75 (HI. 1, 94.5) (97.3, 99.4) (91.7, 93.73 (66.8, 5) (87.5, 97.09 (97 99.4) (92.0) 97.4) (95.0, 9880) (74.0) 92.5)
(88.0, 96.5)
(99.6, 90.1) (96.7, 97.9) (99.4, 85.5) (926, 85.4) 196.1, 74.01 (92.4, 98.3) (99.4, 98.9) (99.8, 92.91 (98.8, 87.9) 1971.5, 91. (97-22, 90.2)
ML.
(99.6, 98.7) (97.0, 91.6 (99.4, 97.6) (93.8, 88.4) 195.7, 83.5) (93.9, 78.8) (99.5, 98.5) (99.8, 99.3) (98.6, 90.60 198.1, 90.7) (97.5, 91.8)
Three
(99.7, 98.8) (97.2, 91.1) (99.5, 98.0) 4, 90-00) (95.9, 86.3) (93.8, 77.1) (99.6, 98.6) (99.8, 99.3) (98.6, 91.99 198. 89.07 (97.8, 92.0)
19

---

RGB
Point score map
Multimodal score map
Ground truth
score map
Figure 10: Failure case in PointAD,
Table 16: Effect point density gap.
Downeampl
Point detection
Mulimodal detecsion
ratio
Global
Local
Global
Local
100%
[820) 94.2)
09555
(86.9.96
6772 90.2)
(85
50%
(79.6,
994.7.8.88)
(84.9) 95.5)
(XG8, 89.9)
30%
(76.6, 91.3
(94.5 79.9)
(83.5. 95.1)
(9044, 89.5)
20%
(72.7, 90.5)
(94.2, 78.2)
(82.0) 94.6)
(95.2, 88.6)
G
Failure Cases
In this section, we present the failure cases of PointAD, which we attribute to direct multimodality fusion. Since our model uses hybrid loss to incorporate the 3D and 2D anomaly semantics, it performs ZS multimodality 3D anomaly detection in a houg-and--lyy manner. However, when one modality prediction deviates severely from the ground truth in rare instances, direct fusion may result in an unpromising multimodal score map. As shown in Figure TO, the hole in the cookie is visually similar to the chocolate on cookies, making it challenging to differentiate the hole anomaly via color information alone. Although PointAD can detect the hole based on its abnormal point relations, the RGB score map heavily influences the final multimodal score map. Conversely. the tire presents an inverse situation where RGB can effectively predict the anomalies, but the point score map fails to recognize it. The false detection could arise from unusual point density and distribution. To demonstrate the effect of point density, we randomly select normal regions of point clouds and subsequently increase or decrease the density of these regions through upsampling and downsampling. We provide a qualitative analysis in II] The visualization shows that PointAD effectively resists noise reasonable levels. However, when noise levels are extremely low high, the corresponding regions become excessively sparse or dense. This causes normal regions to appear similar to hole anomalies squeezed anomalies, leading PointAD to classify these noisy areas anomalies. conmparaettric score alignment and filter methods could be a potential direction, which we leave for further work.
10%
30%
100% (origina)
1000%
3000%
10%
30%
100% (original)
1000%
3000%
Ground Truth
Ground Truth
Figure 11: The impact of noise level. We randomly downsample and upsample part of normal regions to create different point densities.
Robustness to point density To investigate the impact of point density differences between the training and test datasets, we train PointAD using high-density point clouds (original dataset) and then test it on the low-density versions of the datasets, downsampled as described in Input resolution. Table 16 shows that PointAD can still detect anomalies even when retaining 50% of the points from the original point clouds. However, when more points are removed (30% and 20% sample ratio), PointAD experiences an obvious performance degradation. We attribute the misdetection to the overly sparse point clouds forming holes. Nevertheless, PointAD can still detect anomalies even with
20

---

Table 17: Comparison of computation overhead with SOTA approaches on MVTec3D-AD, The unsupervised method is abbreviated as Un.
GPU menory
Point detection
Multimodal deterrion
Methods
Inference time
PP5
usagge (Peak)
BEF (Un.)
0.18
5.50
1924
(76.3.9188)
(97.69923)
(893,, 96.7)
199.5, 97.1)
SDM (Lin.)
0.14
7:14
2716
(96.7. 90.9)
(97.0. 91.8)
(92.1. 97.6)
(93.3, 98,
MEDM (Un.)
98.4)
(93.2,
98.7)
(97.6.
(xx)
3DSR (Un.)
0.09
11:11
3067
(95.1. 94.3)
(93.8. 91.2)
(97.3, 98.6)
(99.3, 97.
PoinCCLPP V2 (2S)
1.52
01.66
9747MB
(78.3, 49.4)
(87.4, 52.3)
(49.8, 79.5)
(51.2, 80.1)
CLIP (25)
3683MB
(4,5444
86.4)
Chenghaan (2S)
81.7)
(88.2.
(-.-)
(.,
WinCLIP (2S)
0.29
3.45
3914MB
(45.2 77.9
(85.8) 59.4)
(58.7) 74.1)
(87.5, 64.2)
AnomalyCI IP (75)
0.19
5.26
3345MR
(36.4.33.5 83.5)
(8..9.009)
666622 87
(91.6 70.9)
4275MB
99..2
significant gap in point density between the training and test domains (c.g., 100% vs. 20%). This demonstrates PointAD's s ability to generalize across different point densitics.
H
Complexity analysis
In Table 17 provide comparison of computation overhead among unsupervised and zero-shot manners The evaluation includes inference time per image, frames per second (FPS), and GPU memory consumption with batch size of 1. For fair comparison, we keep NVIDIA RTX 3090 24GB GPU free until we conduct experiments. Compared to PointCLIP V2. our model requires less time infer an image, achieving higher FPS (2.52 vs. 0.66) with lower graphic memory usage (as discussed in Section 3.5). While CLIP + rendering has slight advantage in computation overhead, our detection performance significantly outperforms Therefore, PointAD achieves favorable trade-off between performance and computation overhead.
Point
Multimodal
Multimodal
Ground truth
score map
score map
truth
score map score map
PointCLIPV2
Figure 12: Visualization comparison between PointAD and PointCLIP V2.
Visualization Comparison
To provide intuitive results, we compare the visualization of PointAD with PointCLIP V2. In Figure 12 our model achieves accurate 7S 3D detection through the point cloud. Moreover, given RGB counterparts, PointAD further improves its detection capacity in M3D detection. However, PointCLIP V2 exhibits noisy activations for normal regions. After incorporating RGB information, PointCLIP V2 appears to struggle to fuse these two modalities in a plug-and-play manner, unlike PointAD.
Additional Visualization
We respectively visualize the 2D renderings and corresponding 2D ground truths, which are rendered from 3D pint clouds and ground truths, as shown Figure [3 We also supplement more zero-shot segmentation results of PointAD in Figure 14
"Since the official code for WinCLIP is not available, we reproduce its results using the implementation found
21

---

View
View
View
View
View
View
View
View
View
renderings
ground truth
renderings
ground truth
2D renderings
20 ground
renderings
ground truth
Figure 13: Visualization about 2D renderings and ground truth from different views (K 9).
K
Detailed results
We provide the class-level 7S 3D results on MVTee3D-AAD Table 18
We provide the class-level ZS M3D results MVTec3D-AD Table
19
We provide the class-level ZS 3D results Eyecandies in Table 20
We provide the class-level ZS M3D results on Eyecandies in Table 21
We provide the class-level ZS 3D results on Real3D-AD in Table 22
We provide class-level ZS cross-dataset 3D results on Eyecandies from MVTe<3D-AD
in Table 23
We provide the class-level ZS cross-dataset 3D results on Eyecandies from MVTec3D-AD in Table
We provide the class-level ZS cross-dataset 3D results Real3D-AD from MVTec3D-AD in Table
22

---

Ground truth
map
Multimodal score map
score map
Multimodal score map
Ground truth
Point map
score map
truth
Point map
Multimodal map
Figure 14: Visualization of point and multimodal score maps in PointAD, which is pre-trained on cookie object.
Table 18: Performance comparison on 3D anomaly detection. The best and second-best results in ZS are highlighted in red and blue. G. and L. represent the global and local anomaly detection.
Mathod
Ragel
glund
Dowel
Repe
CLIP R.
(53.4, 55.2)
(29.) 83.2)
(62.9 89.9
(65.0) $5.01 (65.3, 39.3) (53.0. 78.9)
(72.0) 39.2)
(58.5
83.4)
(80
90.7)
(52
80.1)
(61.2. 35.3)
Cheraghian
(49.3, 80.5)
(52.7,
(43.3,
(59.7.
85.2)
(59
80.5)
(53.6.
PainCLIP V2
PoinC1IP V2.
(47.1, 80.7
(55.1. 84.7)
(47.7. 808) (50.1, 79.8
(50.9 82.) (57.4 83.7)
(52.4. 83.5) (48 78.5)
(54.8
74.4)
(47.8 76.8)
(51.1. 30.6)
AnomalyCLIP
(62.8.
96.91 (51.2 82.2)
(51.9, 54.4) (61.9, 96.2)
(50.0, 90.31 (42.4. 30.1) (69.4. 90.8)
(61.5 85.7)
(62.A. 81.5)
(47.8 77.1)
(56.4, 83.5)
Mupp
(98.3, 99.6)
(53.8, 85.7)
(93.2. 98.5) (89.5. 97.11
(66.3, 59.5) (70.3.91.3
(89.1. 96.9)
(97.8. 99.5)
(91.1. 96.5)
(59.2.
83.7) (80.9, 93.9)
PounAD
82.1)
94.2)
(2.22.)
Cheraghian
(88.2.57.0)
1.
PoinCLIP V2 (78.2, 36.00 (90.9 63.7)
(96.4, 76.0)
(14.7, 24.4)
(93.8. 60.01
(1..) 18.9)
(6.00
43.4)
(92.8
60.1)
(95.
71.7)
(90. 63.3)
(87.4.52.3)
PoinCLIP V2.,
(79.4,35
38.0) (91.5. 68.0)
(9.33.
(34.7.23.)
(93.7. 63.3)
(73.0 16.0
(862
44.0)
(92.7. 58.8)
(95. 69.5)
(904 64.1)
52.3)
Anomaly CLPP (86.0.
69.0)
(69.1.59.7)
(91.4, 73.6)
(79.7.
993.4 73.5)
(7..3.31.2)
(1.)
(93.6
75.6)
(96.7 84.3)
089 60.7)
(88.9, 60.9)
PaDp
94.7)
78.5)
95.0)
97.9)
75.2)
PomnAD
(95-4, 96.9 (93.5, 79.3) (9949,964)
(87.5, 75:4)
(95.5,
75.23
(86.5, 54.1)
(993,
98.3)
(99.)
99.1)
(993)
89.9)
(95.3 79.7) (4.15
23

---

Table 19: Performance comparison on ZS M3D anomaly detection.
Method
Bagel
Cable gland
Carrot
Cookie
Dowel
Foam
Peach
Potato
Roge
Tire
Mean
CLIF+R R.
(5..1,85.9
(55.0, 61.1)
(61,5,90) 90.D (5..)83.)
(9.)1)866)
(6.00,90.7)
(72.0, 91.3
(5677.853
(70.8, 56.01
(5.77
82.91
[60.4,6.4)
MGG
Cheraghian
V2
83.7)
(6.r)
6638,8870 87.6) (47.7
(.-))
(x-2)
(4788
(--)
(513, 80.5) (4.22
6.-9
6.-)
(4922
6.-
(55.4,
6.-
(39.1,6..0
6.-)
6.-
769)
(..)
79.3)
PoinCCLIP V2a
(534, 84.4) (64.7, 89.1 (480. 83.9) (4844 78.4) (47.1. 81.1) (43.9 79.0
(19.6 79.2)
(53.5,
8599
(34.9
60.5
(46.1
76.9
(49.4. 79.8)
AnomalyCT.P
(78.8, 93.5) (58.1, 84.0) (63.2 88.7) (72 89.1) (53.8 82.5) (65. 89.5)
(73.7 91.1)
(64.3. 85.8
(77.5. 10.1) (55.2 52.51
(66. 2, 87.6)
PoinAAD (98 8, 98.7) (74.6, 93.0) (90.0 97.5) (88 96.3) (66.2. 86.9) (79.8, 91.60
(90.8 90.73
(83.1. 93.7)
(93.6. 97.6 068.7. 69
(83.4. 94.9)
PuinIAD (98.8 99.7) (79.9, 91.7)
(95.5, (56
(98.5
(564,
(96.6
(93.6. 97.3)
ClIP R
(.17.9)
(-.8.5)
(.89.5)
(.447)
(-74.3)
(222.1)
(-.4755
(-.22.7
(-:73,)
(-.8%9)
(- (6.660)
ML.
Cheraghian
(.-)
(.:)
(.:)
6.3
6-4
6.-1
6.
6-
[.
V2 (40.6 78.0) (56.1. (5.44 (538. 84.2) (52.7 8LD) (50.7 80.4)
(507.80.4) (4088 78.1)
(54..
62.8
(48.9.
77.9)
(54.3
72.5)
(59.3
9)
(78.3. 49.4)
PrinCTIPP v2, (75.9. 40.8) (76.2. 47.4) (92.5, 79.9) (7177 80.7) (72.8 44.9)
(12.3.21
(77.1
46.4)
(874
63.7)
(87.9
60
990.8
70.81
(79.5, 51.6)
AnomalyCLIP (93.7, 71.1) (50.7.67) 67.7) (95. 84.7) (8200 (5.2) (919. 72.1 (84.3 50.
69.5
79.)
(85.6 63.0)
995.9. 83.0)
99.. 67.5)
70.9)
PolxAAD-ooOp (9.9, (95.7, 87.3) (99.3,
(910 0. 095.
(918 72.2)
68
7, 96.7)
0994 97.9)
09.66
91
PoinAD
(99.0) 90.1) (96.7, 9739) (99.4, 85.5) (22.0 85.4)
09 75.0
02
4,
98.9
09
96.9
1993.
92.1
198.8 97.9
097.5
91
(/712 90:21
Table 20: Performance comparison on ZS 3D anomaly detection on Eyecandies.
Method
Candy
Cane
Chrcolate
Cookse
Chocolate Praline
Contetto
Gummy Bear
Hazelout Trulle
Licorice Sandwich
Lollipop
millow
Marsh-
Peppermint
Candy
Mesn
CLIP Rendering
(61 8, 60.9)
(48.3, 53.8)
(61.8, 72.0)
(82 1, 86.9)
(8) 4, 83.7)
(57.3 $4.4)
(72.6. 71.9)
(66.2. 55.8)
660.6 68.9)
(75.4. 83.9)
(66.7, 69.2)
Chenghian
(5000 50.0)
(50.D, 50.0)
(50.0) 50.0)
(58.0 50.00
(500, 48.0)
(50.0) 50.00
(50.0. 50.0)
(46.7.
50.80
(48.0
52.13
(50.0, 50.00
(49.5, 48.1)
G.
PoinCLIPV2
(45 1, 48.9)
[55.5, 54.7)
(37.3, 42.1)
(30.9) 40.3)
(33.5, 42.4)
(40.0. 45.0)
(67.0. 02.7)
(41.2, 28.1)
(54.1, 55.6)
(56.0) 63.5)
(46.1, 48.1)
PoinCCIPP
(45
40.4)
41.0)
(50.9. 58.
10.9)
7467.47.9) (99.1,57.5)
(44.4, 47.0)
AnvesalyCT.IP
47.7.
(34.9.4 42.0)
(57.5, 62.0)
(74.3,
76.3)
(40 52.2)
000.8 70.7)
(52.6,
(60.1,
46.2)
(64.0, 62.4)
(68.4. 72.7)
(57.6 59.00
PuimIAD-CaOp
(8.17.48.1)
(56. 61.44
(72.6, 82.8)
(82.5.87.8)
(66.2. 71.8)
(60.8 63.9)
(80.6, 84.5)
(70.5,
62.8)
(54.1. 69.7)
(77.9, 85.3)
(67.7. 71.8)
Point-AD
(428. 51.00
(51.2, 55.4)
(75.1, 84.00
(8.2,
(201) 78.7)
(99.9 61.3)
(81.8) 85.5)
(80.1,
75:2)
(68.3 73.35
(50.4, 85:4)
(69.1.73.8)
CLJP Rendering
(97.3, 84.0)
(77.7, (7.22.243)
(71.0, 19.2)
(76.7 25.6)
(84.5, 37.4)
(76.5, 33.9)
(79.3, 50.9)
(94.6, 66.6)
(709,
(83.1, 39.4)
(81.2, 37.9)
Chenghian
(6.-)
(.,-)
(S--)
PoinCLiPV2
(45.0.-
(38..))
(48.9-)
(43.3.)
(45.0.-)
(54.3. 19.6)
(386.-)
(43.4.)
(42.3
(37.9.-)
(43.7, -)
PoinCLIPVZ,
(45.0)-)
(384.-)
(51.2, 16.1)
(43.5
(45.2,-)
(55.2, (1.22,21.))
(38.8 -)
(43.4.))
(43.1.157) 15.7)
(37.9
(44.2.-
AnomaalLLLPP
(95.9, 84.3)
(73.4, 32.2)
(79.3, 43.0)
(14.6 37.7)
(7877
(08.9, 26.1)
(76.9, 58.4)
(92.0.1.3
(58.5 16.2)
(7.1, 39.1)
(TT. 43.4)
PaimAD--CaDp
(98
68.7)
69.2)
44.5)
(86 64.00
(92.3,
71.3)
Point-AD
(98 87.8)
(92.9, 74.8]
(90.9) 65.7)
(943) (8.2)
(8877 71.4)
(54.6 45.1)
(93.8 72.0)
(97.8 86.0)
(873 64:4)
(92.6, 76.00
Table 21: Performance comparison on ZS M3D anomaly detection on Eyecandies.
Method
Candy
Cane
Chocolate
Cookie
Chocolate Praline
Conferto
Gummy Bear
Hazelout Truille
Licarice Sandwich
Lollipop
mallow
Massh
Peppermint Candy
Mean
CLIP + Rendering
(64.3, 67.8)
(76.6, 77.6)
(64.3, 70.8)
(88.0 89.6)
(55.5 (2.1)
(18.4
(7.75,677)
77.8)
(83.4 62.7)
(73.0, 73.9)
Cherighian
(..))
(c.-)
(.-))
(.-)
(6-))
(--))
(--)
[--]
(---)
MG.
PoinCLIPV2
(43.0,
55.0)
(16..55..)
(49.3.484)
(9.17.9.1)
(5.3
(61.8, 70.0
(5.1. (02.1.30.4
(1122
(169.
PoinCI.PPV2,
(44.1. 51.2)
(44.5, 52.4)
(48.. 52.3)
(56.7,443) 54.8
(42884448)
(557.603 60.3)
(43.8 (63.5.687))
(2.1
(540,
(31.3,39
(48.5. 50.5)
AsomalyC1.PP
(49.7. 50.8)
(2.9).
(66.5, 70.3)
(66.7,680)
(64.0 69.5)
(61.1
(.9.2
(6.5
(773,8044
(6.65.74.)
67.5)
MiimAD-CoOp
(39.4.45 45.6)
(81.6, 87.2)
(84.0, 87.7)
(886. 91.1)
(66.5 69.2)
(64.3.692))
(1.77880))
(61.953.0)
(800, 83.5)
(54.7. 89.6)
(73.7. 76.0)
Point-AD
49.1)
(87.2
(863,
C IP + Rendering
(97.3,89.2)
(72.8, 10.3)
(65.5.8.3)
(750, 17.4)
(83.9 31.8)
(72.4, 29.5
(75.6,
(94.4, 75.8)
(66.3, 15.7)
(7677
(780,
Cherighian
[6-)
(-.-)
(..-)
(---)
(..))
(5--)
(s--)
(6-))
(.-))
MT.
PoinCLIPV2
(44.8.-
(44.8-)
(48.0.) -)
(59.6-)
(48.6-)
(53.9-)
(42.2-)
(33.7.)
(43.5.)
-)
(46.0.-)
PoinCL.1P2,,
(44.8.-
(44.7-) -)
(42.0.
(9).3.-
(48.2
(54.2-)
(42.2-)
(33.6.)
(45.0.)
(41,5.-
(46.2.-)
AsomalyULIP
88.5)
64.3)
(53.9.554))
(92.1.74.)
53.1)
(00.2,273)
52.9)
(94.6
38.2)
(848, 48.8)
PoimAAD-CoOp
(97.5 5, 91.1)
(9688 97
(94.8, 85.2)
(98.4 91.7)
(85.9
57.3)
61-1)
(97.3, 89.0)
(96.6, 86.3)
(96.6. 07.1)
87.5)
(94.0) 87.6)
Point-AD
(964, 87.2) 197.8 90.31
83.75
94.77
(908, 74.4)
(87.7) 62:5)
(96.8, 58.0)
(96.7, 85.4)
(97.) 88.5)
(27.1, 85.0)
(95.3 84.3)
Table 22: Performance comparison on 7S 3D anomaly detection on Real3D-AD.
Method
airplane
candybar
chickas
dianond
duck
genvatane
scahouse
shell
starfish
toffoes
Meas
Mendering
(6199.569)
58.1)
(73.5, 20.5)
(59.5, 20.2)
($4, 86)
(T1, 76.21
[61.2, 69.3)
168.3, 69.9)
(83.6, 97.99
(6077 54.5)
169.5, 78.1) (36.9, 89.6)
(6888 72.3)
Cherughian
(52.5, (57.8.57.3)
(45.3.56.. (1.33
(41.9, 47.3)
(47.8, 56.3)
(49.0, 69.9)
(50.2
53.91
(45.6.
(56.3,
(51.5. 36.71
57.2)
(3.7.5544
(4..3.
G. PoinCLiPV2
(49.9,
(41.7, 48.3)
(448. 49)
55.9)
(51.5 48.9)
(68.4 70.4)
(00.9) 68.0)
(69.6 69.3)
(53.1. 39.4)
(41.9,
52.8
(31.1,
43.3)
(78.6 K2.4)
(53.1,
PoinCLIPY2,
(46.5,
48.1)
49.91
(47.8.48.9)
(00.6
60.81
75.9)
(64.5,
(61.76002)
167.3, 693)
(57.5,
AnomalyCLiP
(61.7.55.4
(51.2, 52.7)
(1.77
(57.9.67.)
67.7)
(65.0.65.2
(56.2, 59.1)
(56.4,
50.8)
(5..5,57.6)
(53.1.49.7)
(51.8.58 58.7)
(51.0, 52.3)
(55.2.57.1)
PointAD-CoOp
(60.8, 59.5)
(08.7. 71.91 (15.1,
78.6)
(45.3, 59.1)
(98.A 98.5)
(54.4, 47.5)
(16.3, 78.9
(56.6, 87.00
(81.4.
78.9)
(89.4, 88.8
(53.5, 53.9)
(67.1. 75.1)
(75.9, 75.9)
PD
(0..9)010) 11399 72:4)
(1L., 76.81
(5200) 54,2)
(99.2 99.2)
(60.1, 62.3)
(94.3, 78.7)
(87.3, 87.50
(7699) 81.1)
(89.5, 88.9)
(80.9) 87.2)
(09.0) 72:3)
(748, 76.99
CLIP Rendering
(18.4-)
(48.-
(33.7--)
(47.L-)
(316,) -)
(19.6.)
(56.5.
(50.3. -)
(34.7.
(47.5
(54. -)
(49.2 -)
Cherighaa
[.-)
6.4
6--
(2-)
(.-)
6.-)
(6-9
(.-)
(.-)
L. PoinCLiPV2
(4
(56..-)
(55.2.))
(46.5
(52.6
(62.6
(639.-)
(51.5.-)
(48.3.)
(S8.1
(40.6 -)
(53.6.)
(S29,
PoinCLIPV2,
(69.6-)
(5t.3-)
(56.2.
(47.1.)
(53.2-) )
(616.))
(592.-)
(51.6.
(47.2.
(59.3.
(41.2.) -)
(68.1.)
(522.
AnomalyCLII
(51.1.)
(488.-)
(51.7-1
(50.0.
(55.2.
(48.9-)
(46.5.)
(48.9-1
(49.2.-)
(50.6
(51.0.)
(51.).-
(50.3
PointAD-CoOp
(6
(
(67.1.)
(64.2.
(87.2.)
(508.)
(79.1.)
(81.7.)
(77.0,
(73.3.
(77.6
(68.4)
026,
Point-D
(
(71.33
(61.7
(573.)
(5100
(60.1.)
(80.2.
(718.
(7.8.3
(
(700L-)
(23.5
24

---

Table 23: Perfromance comparison on ZS 3D cross-dataset anomaly detection transferring from MVTec3D-AD to Eyecandies.
Method
Candy
Chocolase Cookie
Chocolate Praline
Contetto
Huzelaut Trufile
Licorice Sundwich
Lotlipop
Mansh mallow
Peppermint Candy
PoinCLIPV2,
55.6
(318
(38.6.
(3.00 40.3)
(54.5 54.3)
(42.7. 30.4)
(58.8,
(47.7,
(85.2.48.)
AnommalyC1.IP
(49.6, 52.0)
(4311.8) 48.1)
(652, 67.3)
(0955 7044
(41.2. 44.6)
(53.8 56.6)
(51.2, 55.0)
(604, 4544
(58.1,58.3)
(70.6 73.3)
(5..3.57.1)
PoinAD-Cool
(54.5, 58.5)
(51.6 59.13
(7288 81.))
(79.3, 86.4)
(69.1.76.8
(62.6 61.3)
(78.9,83.6 83.6)
(74.466.5)
(67.2 73.7)
(69.1.73.8) (81.1.870)
Point-AD
(31.7, 58.99
(5799 64.75
(768, 84.4)
(79.9 87
(66.5. 75.0)
(61.5,61 61.66
(8).0
07.3
(3.3,68.55 63..6855)
(79.5, 56.0)
(69.5, 74
PoinCT Po
(45.0.-)
(3844-)
(49.2.-)
(43.2
(45.0-)
(55.2 21.6)
(38.7-)
(43.4-)
(42.9-)
(37.9.))
(43.9.)
AnsmalyCLIPP
(763.34.1)
(80.1.4677
(7..7.44.))
(80.4,42.7) 42.7)
(735,33.1)
(80.740088)
1908,
(79.6.454) (794.45.0)
ArirrAD
(97.5, 85.7)
(9244 74.4)
(02.1, 70.6)
(04.0)
(89.0, 60.3)
(84-4 46.5)
(93.1. 72.7)
(97.8 86-8)
(86.1,50 599-4)
(92.0. 71.8)
(91.8 70.5)
Fain-AD
(97.5, 85.4) (9522 26.89
(919) 70.2)
(14.5 72.31
(88.5 68.3)
(82.6 45.4)
(9.0)
(98.0) 89.0)
(85.8 58.9)
(3.3)
(9LX, 71.4)
Table 24: Perfromance comparison on ZS M3D cross- anomaly detection transferring from MVTec3D-AD to Eyecandies,
Method
Chocolate Cookie
Chocolate Praline
Confetto
Gummy Rear
Hazelnut Truffle
Licorior Sandwich
Lollipop
Mamh- mallow
Peppermint Candy
Mona
MG.
AnomalyCLIP
PoisCLIPP2,,
(1.4)
(42.1, 50.4)
(45.5 54.9)
(66.1, 68.9)
(49.0, 52.5)
(664, 77.7
(79.3, 82.7
(57.1,
(44.9, 45.8)
310056677
(53.6 56.6)
(S5.1, 58.3)
(59.0, 64.5)
(73.2, 79.6)
(47.4, 56.1)
165.4, 57.9)
(70.3.2255) 72.5)
(5.11,5333)
(32.6
(72.8 75.5)
(48.5, 50.9)
(68.7, 68.1)
ProOrADDCrOOp
(53.4 58.6)
(728, 80.4)
(83.6, 86.6)
(73.9, (2.81
92.8)
76.2)
(72.)) 76.23
(94.6, 88.4)
(65.7. 54.3)
(78.4, 83.3)
(89.6 92.7)
(76.3, 78.9)
Pain-AD
MI.
Poin
AnIMalyCLIP
(9..1.82.3)
(45.1.)
(47.2,
71.2)
(84.3, 62.0)
(48.1-)
(94.), 90.9)
(63.1
(80.6.44. 44.5)
(49.7.)
(73.6, (2.7)
(54.0) 15.6)
(89.6, 66.9)
(45.2.)
(92.6,
(33.8
(75.9,
(44.7-1
(86.2, (21.9.5.11)
(41.9.)
(47.3.
PoorADDCCOOp
(92.0) 71.0)
(96.8 86.4)
(94.2, 86:2)
(97.9 92.55
(87.9, 64.8)
(91.3, 68.6)
(96.2, 83.60
196.5, 82.5)
(95.3, 81.3)
(96.1, 86.7)
(94.4, 80.3)
Pount AD
(88.8, 63.9)
(97.4, 88.9)
(92.5, 85.3)
(99.1,90)
(8933) 70.4)
(8959 70.43
(95.5, 85.4)
195.7, 79.8)
(95.8, 82.5)
(9522, 84.8)
(94,0) 30.7
Table 25: Perfromance comparison on ZS cross-dataset anomaly detection transferring from MVTec3D-AD to Real3D-AD.
Method
nirplane
candyber
chicken
disnoodd
gontstonc
scahorse
starfish
soffecs
PoinCLIPV2,
46.4) (47.0.
(49.4,
57.1)
(47.6..00 (..00.
(64.6 62.1)
(60.3, 67.4)
(60.2,
(60.5,
(8044 (6.55
(72.3,
(57.4 (8.84
AnomalyCLIP
(1.88
(6333,77.3 67.7)
(48.8, 50.1)
(51.9, 62.0)
(60.2, 29.2)
(42.6. 46.8)
(57.1,587)
154.6, 58.21
(00.0.001)
(51.1.50.1)
(5.6)
(53.8, 58.0)
(52.7.35.7)
PoinAD CoOp
(55.2, 55.7)
(61.0) 66.2)
(68.3, 68.7)
(57.2, 58.61
(99.3, 99.3)
(75.0, 73.4)
(78.1, 82.4)
(85.8, R5.5)
(73.5, 78.2)
(89.8, 36.8)
(76.1. 81.4)
(75.6, 76.2)
(74.8 76.99
Paint-AD
62.8)
99.0)
67.2)
(74.5, 79.4)
(8977
(7.9)
PoinCT JPV2,
PPV2
(50.5.
(53.6,
(54.2-))
(47.0-)
(S4.1-)
(60.7.)
(59.4, -)
(51.2,)
(47
(554, 18.2)
(41.6.
(47.5.
(51.9.-)
AnomalyCLIP
(50.9.
(19.6.)
(19.8.)
(50.1.)
(57.5.)
(18.6.
(502
(50.5,
(19.3.
(51.0-)
(50.3,
(61.4. -)
(71.2.-)
(64.6-)
(67.7.-
(85.0 -)
(5
(76.9.-)
(780.-)
(77.3. -)
(59.7.-)
(75.2-)
(70.1.-)
Point AD
(61..
(69.7.)
(004.)
(673.
(86.6 )
(50.3, -
(75.6.)
(789.)
C
08.11
(66.8,
(76.4
(2L.6
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction precisely reflect the contribution and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: have created separate "Limitations" section in our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed illustration our proposed algorithm and baselines in the Appendix C
Appendix B and Appendix
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [No]
Justification: we will make our code and dataset available once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Policyylly for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc///uuii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide full details in Appendix and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the average results across three runs in Section 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We point out the specific compute resources in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://e http://ppw....pp...c.//cccccttaa
Answer: [Yes]
Justification: Our paper obeys the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we respect the Licenses for existing assets that we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsippteeeee www.reeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
30

---

Answer: [NA]
Justification:
we will release new assets proposed in our paper once the paper is accepted.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "edPTVSza3p", "reviewer": "Reviewer_1QVN", "summary": "The paper propose a unified framework, PointAD,to detect 3D anomalies in a ZS manner. Hybrid representation learning is proposed to incorporate the generic normality and abnormality semantics into PointAD. PointAD can incorporate 2D RGB information in a plug-and-play manner for testing, which can perform ZS M3D anomaly detection directly. Experiments results demonstrate the superiority of the model in detecting and segmenting 3D anomalies.", "questions": "See weaknesses.", "limitations": "The authors adequately addressd the limitations. Societal impact is not discussed, but I believe it's not needed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PointAD introduces a novel method for Zero-Shot 3D anomaly detection by integrating the strong generalization capabilities of CLIP to recognize 3D anomalies from unseen objects. This approach is first of its kind in ZS 3D anomaly detection.\n2. The paper describes a unified framework that comprehends 3D anomalies from both points and pixels, allowing for a comprehensive understanding of anomalies by rendering 3D objects into 2D renderings and projecting them back to 3D space. It proposes hybrid representation learning to optimize learnable text prompts from both 3D and 2D data, enhancing the model's ability to generalize across diverse unseen objects.\n3. Extensive experiments demonstrate the superiority of PointAD over existing state-of-the-art methods, especially in ZS settings where traditional models struggle.", "weaknesses": "I do not notice any obvious weaknesses in this paper, but I still have a few questions that I hope the authors can answer.\n1. The methodology, while innovative, is complex due to the need to render 3D objects into multiple 2D views and then re-project them, the performance of the model may heavily depends on the quality of the 2D renderings of the 3D objects, Therefore, if the renderings are of poor quality, could the model's performance degrade?\n2. The experiments in the paper utilize high-resolution point cloud scans. Given that the point clouds obtained in practical applications may not always be high resolution, how effective is this method in detecting anomalies in low-resolution, sparse scans?\n3. In the appendix's failure case section, the detection of the tire failed on the point score map. Thus, if a tire, or any object, has noise, how significantly does it affect the recognition on the point score map? I suspect that the possible reason for this failure could be the algorithm's sensitivity to point density and spatial distribution. When anomalies are located in areas where point density or distribution are inconsistent or significantly different from the training data, can this model still generalize well to new, unseen anomalies?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 1QVN\n\nThank you very much for your acknowledgment and insightful comments. Your comments inspire us to explore the robustness of PointAD from the rendering process and input under different conditions.\n\n**Q1: If the renderings are of poor quality, could the model's performance degrade?**\n\nThank you for pointing out your concerns. PointAD interprets point clouds through their corresponding 2D renderings, and the quality of these renderings impacts the information that PointAD can extract from the original point clouds. In our manuscript, we used the Open3D Library to render the point clouds, but it does not provide an API for controlling rendering quality. To simulate varying rendering quality, we applied Gaussian blur with different extents ($\\sigma$) to the 2D renderings. Sample visualizations are included in `Figure 1`. Specifically, we conducted experiments on MVTec3D-AD using different blur sigma values (i.e., $\\{1, 5, 9\\}$). `Table 1` shows that the detection performance of PointAD diminishes as rendering quality decreases (with increasing sigma). However, the degradation is acceptable even when the renderings are heavily blurred (sigma equals 9). In such cases, PointAD still outperforms baselines that use high-quality renderings.\n\n**Q2: How effective is this method in detecting anomalies in low-resolution, sparse scans?**\n\nWe appreciate your detailed feedback, which has been invaluable in refining our work. To create low-resolution point clouds, we downsample the entire high-resolution point clouds using Farthest Point Sampling (FPS) with various sampling ratios. This strategy allows us to generate corresponding low-resolution datasets for training and evaluating PointAD. Visualizations of these datasets are provided in `Figure 2`. We train PointAD using the resulting low-resolution samples and test PointAD on the same resolution.\n\n`Table 2` demonstrates that PointAD maintains a strong detection capacity for low-resolution point clouds when the downsampling ratios are 20\\%, 30\\%, 50\\%, and 70\\%. Even at 20\\% resolution, PointAD still achieves state-of-the-art performance. This indicates that PointAD is generally applicable to point clouds with various resolutions.\n\n**Q3: How significantly does the noise affect the recognition on the point score map?**\n\nThanks for your insightful comments. We agree that failure detection could arise from unusual point density and distribution. To demonstrate the effect of point density, we randomly select normal regions of point clouds and subsequently increase or decrease the density of these regions through upsampling and downsampling. We provide a qualitative analysis in `Figure 3`. The visualization shows that PointAD effectively resists noise at reasonable levels. However, when noise levels are extremely low or high, the corresponding regions become excessively sparse or dense. This causes normal regions to appear similar to hole anomalies or squeezed anomalies, leading PointAD to classify these noisy areas as anomalies.\n\n**Q4: When anomalies are located in areas where point density or distribution are inconsistent or significantly different from the training data, can this model still generalize well to new, unseen anomalies?**\n\nTo investigate the impact of point density differences between the training and test datasets, we train PointAD using high-density point clouds (original dataset) and then test it on the low-density versions of the datasets, downsampled as described in **Q2**. `Table 3` shows that PointAD can still detect anomalies even when retaining 50\\% of the points from the original point clouds. However, when more points are removed (30\\% and 20\\% sample ratio), PointAD experiences an obvious performance degradation. We attribute the misdetection to the overly sparse point clouds forming holes. Nevertheless, PointAD can still detect anomalies even with a significant gap in point density between the training and test domains (e.g., 100\\% vs. 20\\%). This demonstrates PointAD's ability to generalize across different point densities.\n\n**In summary, the experimental results demonstrate PointAD's robustness to diverse rendering conditions. We attribute this robustness and generalization to PointAD's efficient hybrid representation learning and CLIP's powerful feature extraction. If you are satisfied with our responses, we would greatly appreciate it if you could consider raising your score. Your support is very important to us.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfUyrocIiX", "reviewer": "Reviewer_GgEU", "summary": "The paper introduces PointAD, the first approach to explore the domain of zero-shot 3D anomaly detection, leveraging CLIP's strong generalization to identify anomalies in previously unseen objects. It offers a unified framework that integrates 3D point clouds with 2D renderings, employing hybrid representation learning to capture the semantics of anomalies. PointAD's key contributions are its pioneering exploration of the ZS 3D anomaly detection domain, its ability to seamlessly integrate multimodal data like RGB information for enhanced detection, and its superior performance over existing methods. The robustness of the framework is confirmed through extensive experiments.", "questions": "1.\tPlease answer the main differences with these mentioned methods in Weaknesses and compare with part of them on experiments.\n2.\tGiven the concerns raised in the Weaknesses section 1, does PointAD still hold the merit of computational efficiency when compared to other methods such as AnomalyCLIP and Cheraghian?\n3.\tCould you elaborate on the multi-view rendering process of point clouds (as described from line 134 to line 140), particularly how the color of the rendered images is determined? Furthermore, at line 254, it's stated that additional RGB information is utilized only in the test dataset. Given the description at line 564, which refers to 2D RGB information, how is this 2D data integrated with the rendered multi-view images? Are there any strategies which are employed to bridge the gap between the rendered training data and the RGB information used in testing?", "limitations": "The authors have discussed the limitations of their model, and there are no potential negative societal impacts associated with their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces PointAD, a novel method in 3D anomaly detection that leverages CLIP for 3D analysis, uniquely combining point clouds and pixel data. It expands the application of vision-language models into new domains, showcasing versatility in 3D point cloud analysis.\nSoundness: The paper exhibits methodological soundness through its rigorous experimental setup, including the use of diverse datasets and a thorough ablation study that substantiates the design decisions and effectiveness of the proposed PointAD framework. The state-of-the-art experiments performance confirms the model's outperforming ability.  \nClarity: The paper stands out for its clarity, guiding readers smoothly from the problem statement to the final results. It skillfully explains complex concepts in an accessible way, ensuring that a wider audience can follow along. The paper also benefits from helpful visual aids that clearly demonstrate the model's effectiveness.", "weaknesses": "1.\tThere’s some zero-shot anomaly detection methods with clip on 2D images, so this paper should compare with them, especially, from the view of techniques’ differences, not only the used data differences. For example, those zero-shot methods (WinCLIP, VAND and AnomalyCLIP) mentioned in [R1] and [R2].\n[R1] Cao, Y., Xu, X., Zhang, J., Cheng, Y., Huang, X., Pang, G., & Shen, W. (2024). A survey on visual anomaly detection: Challenge, approach, and prospect. arxiv preprint arxiv:2401.16402.\n[R2] Li, X., Huang, Z., Xue, F., & Zhou, Y. (2024). Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images. arxiv preprint arxiv:2401.16753.\n\n2.\tI also wonder if the new methods of 3D anomaly detection can also be used to deal with this zero-shot task. For example, [R3] and [R4]. \n[R3] Zuo, Z., Dong, J., Wu, Y., Qu, Y., & Wu, Z. (2024). CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation. arxiv preprint arxiv:2406.18941.\n[R4] Li, W., Xu, X., Gu, Y., Zheng, B., Gao, S., & Wu, Y. (2024). Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22207-22216).\n3.\tMore importantly, there’s some very similar studies with this work and task. Authors should mention and compare them within this study. For example, [R5].\n[R5] Li, Y., Goodge, A., Liu, F., & Foo, C. S. (2024). Promptad: Zero-shot anomaly detection using text prompts. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1093-1102).\n[R6] X. Chen, J. Zhang, G. Tian, H. He, W. Zhang, Y. Wang, C. Wang,Y. Wu, and Y. Liu, “Clip-ad: A language-guided staged dual-\npath model for zero-shot anomaly detection,” arXiv preprint\narXiv:2311.00453, 2023.\n[R7] Wang, C., Zhu, H., Peng, J., Wang, Y., Yi, R., Wu, Y., ... & Zhang, J. (2024). M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising. arxiv preprint arxiv:2406.02263.\n4.\tComputational Efficiency: Although the authors have commendably addressed computational challenges, the overall computational cost associated with rendering multi-view images and processing them through the vision encoder is still substantial. Since this field is still developing with few comparable studies, it is recommended that the authors provide a broader comparison about the computational cost with existing anomaly detection techniques, including those like Cheraghian, CPFM and 3DSR that are not strictly within the zero-shot category. This expanded comparison will help to show that the computational demands of PointAD are reasonable and suitable for practical use cases.\n5.\tClarity and Completeness of Presentation: The paper's writing is generally clear and free of errors, yet minor aspects of the presentation could be polished for better reader comprehension. For instance, the figure overlays, specifically in Figure 1, currently only elucidate part (a), leaving explanations for parts (b) and (c) to be found in the main text. This separation can disrupt the reader's flow and understanding. Enhancing the figure overlays to include all relevant parts and ensuring that each component of the figure is self-explanatory will make the paper more accessible and its findings easier to grasp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer GgEU\n\nThanks for acknowledging our work. Your insightful comments help highlight our technological contributions in comparison to concurrent works and further promote comprehensive evaluations. \n\nDue to the character limit, please refer to the bottom of **General Response** for our reply to **Q1**.\n\n**Q2: Please compare with part of them on experiments.**\n\nWe have included these simplified illustrations in the revised version. For experimental comparisons, since the source codes for PromptAD, CLIP3D-AD, IMRNet, and M3DM-NR are not available, we compared PointAD's performance with WinCLIP and VAND (AnomalyCLIP has been incorporated in the current submission). Note that VAND and CLIP-AD are similar methods, and we selected VAND for comparison due to time constraints. `Table 4` shows that PointAD significantly outperforms these baselines across all datasets in the one-vs-rest setting. The consistent superiority across diverse baselines demonstrates the effectiveness of PointAD. Due to the page limitation, we will provide results for the cross-dataset setting in the revised version.\n\n**Q3: Does PointAD still hold the merit of computational efficiency when compared to other methods such as AnomalyCLIP and Cheraghian?**\n\nThanks for pointing out your concern. We have supplemented the computation overhead of WinCLIP, VAND, and AnomalyCLIP. In addition, as you recommended in Weakness Section 2, we report the computation consumption of unsupervised methods such as CPFM and 3DSR. From `Table 5`, we can observe that PointAD consumes medium computation overhead to achieve the best results. A lightweight version of PointAD is a worthwhile direction to explore for future work.\n\n**Q4: This separation can disrupt the reader's flow and understanding.** \n\nThanks for your detailed feedback. We agree that the incomplete figure captions could confuse readers. We will supplement the necessary captions to all figures in the manuscript to enhance clarity.\n\n\n**Q5: Could you elaborate on the multi-view rendering process of point clouds (as described from line 134 to line 140), particularly how the color of the rendered images is determined ?**\n\nGiven the point clouds and their corresponding point-level labels, we first resize both the point clouds and labels to 336 × 336. To represent the 3D point clouds in multiple 2D renderings, we use the Open3D library to render the point clouds from various views in a circular manner to preserve 3D information. We use gray ([0.7, 0.7, 0.7]) as the rendering color for all objects in our experiments for two main reasons: \n1. Prominent colors might introduce a correlation between anomaly semantics and color information, potentially reducing PointAD's generalization capacity. Using gray helps mitigate this color-induced bias in detecting 3D anomalies. \n\n2. Gray effectively represents 3D anomalies in 2D renderings, facilitating PointAD’s ability to capture the corresponding anomaly semantics.\n\nPointAD requires corresponding 2D labels to explicitly align 3D and 2D anomalies. Therefore, we color the 3D point clouds according to the point labels, with normal regions marked as 0 (black, including the background) and abnormal regions as 1 (white). We then render these from the same angles to obtain the corresponding 2D pixel labels.\n\n**Q6: Given the description at line 564, which refers to 2D RGB information, how is this 2D data integrated with the rendered multi-view images? Are there any strategies which are employed to bridge the gap between the rendered training data and the RGB information used in testing?**\n\nThank you for raising your concern. Since 2D RGB data includes the RGB information of all points, we directly feed this 2D data into PointAD to obtain both the RGB anomaly scores (global) and the anomaly score maps (local). As shown in `Figure 4`, the RGB information can be considered as the additional 2D rendering that includes RGB information of all points. Unlike 2D RGB information, 2D renderings from different views represent only partial information. We need to aggregate these renderings to generate the point anomaly scores and point anomaly score maps. For fusion, we combine the RGB point anomaly score and the anomaly score maps using a weighted average.\n\nWe do not use any additional strategies for fusing RGB and point clouds. The effective fusion is due to the explicit alignment of 3D and 2D spaces, which allows PointAD to learn from both point and pixel anomalies collaboratively. This approach captures generic abnormal patterns that are independent of object class semantics and colors, enabling PointAD to generalize well to RGB information during testing. It is worthwhile to explore a fusion mechanism to further bridge this gap. One potential approach could be to introduce 2D rendering information as a condition for representation learning, which might improve PointAD’s generalization to unseen class semantics.\n\nWe appreciate your acknowledgment of our work. If our responses satisfy you, we would be grateful if you could consider improving your scores.\n\n---\n# Due to character limitation, we continue the response to **Q3** for **Reviewer dthJ** here.\n\n**Q3:  Experiments on additional datasets**\n\nSince 3D anomaly detection is an emerging field, publicly available 3D datasets are limited. Fortunately, we have found a newly available industrial dataset, Anomaly-ShapeNet. However, we fail to find a point cloud dataset related to medical diagnostics and plan to incorporate such data when it becomes available.\n\nAnomaly-ShapeNet is a challenging dataset, comprising 1600 point clouds from 40 classes with 6 anomaly types, without RGB information. We conducted a one-vs-rest setting, using ashtray0, bottle, and cup0 separately to train PointAD. The averaged results on the remaining classes are reported in `Table 9`. The results indicate that PointAD consistently achieves superior performance by a significant margin compared to other baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SEBrw5NVwH", "reviewer": "Reviewer_dthJ", "summary": "The paper titled \"PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection\" introduces PointAD, a novel framework designed for zero-shot 3D anomaly detection by leveraging both point clouds and RGB images. The approach builds on the strong generalization capabilities of CLIP and adapts it to 3D anomaly detection. The framework renders 3D anomalies into multiple 2D views and then projects these 2D representations back into 3D space. This hybrid representation learning allows for capturing generic anomaly semantics across various unseen 3D objects. PointAD also incorporates auxiliary point clouds to optimize learnable text prompts for better anomaly detection and segmentation.", "questions": "1. Could the authors provide more insights into the selection of rendering angles and the impact of different numbers of views on the detection performance?\n2. Have the authors considered the robustness of the method under different environmental conditions, such as varying lighting and occlusions?\n3. Could additional experiments on other industrial or medical datasets help to further validate the generalizability of the proposed method?\n\n\n======== post rebuttal ==========\nThe authors' rebuttal solve most of my concerns, hence I raise my score to borderline accept.", "limitations": "The authors have addressed some limitations, including the fixed rendering angles and the potential for further optimization of view selection. Constructive suggestions for improvement include exploring the sensitivity to rendering parameters and validating the method under different environmental conditions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of CLIP for 3D anomaly detection and the hybrid representation learning approach are novel contributions.\n2. The theoretical foundations are robust, and the empirical validation is comprehensive.\n3. The paper is clearly written, with well-structured sections and effective use of visual aids.\n4. The approach addresses a critical gap in the field, with potential for significant impact on future research and practical applications.", "weaknesses": "1. The sensitivity of the method to the selection of rendering angles and the number of views could be explored further.\n2. While the experimental results are compelling, additional validation on more diverse and challenging datasets would further strengthen the claims.\n3. The discussion of potential limitations and future work could be expanded to provide a more comprehensive view of the method's applicability and areas for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer dthJ\n\nWe appreciate your recognition of our work. Your insightful suggestions help make our sensitivity analyses more comprehensive and convincing. In addition to the response to your valuable feedback, we conducted further explorations on rendering quality and input resolution (**Q1 and Q2 responses to Reviewer 1QVN**).\n\n**Q1:  Insights into the rendering angles and different numbers of views** \n\nPointAD interprets point clouds from their 2D renderings, where rendering angles and numbers collectively determine the amount of information derived. They have different emphases. For rendering angles, the importance lies in the discrepancy between adjacent angles, as this affects the information granularity that PointAD obtains from adjacent views. When the angle discrepancy is fixed, the number of renderings determines the coverage of 3D information in the resulting 2D renderings. To capture all point cloud information, especially abnormal points, it is crucial to ensure comprehensive coverage. Therefore, our approach in selecting rendering angles and the number of renderings is to guarantee that all points in point clouds are adequately represented.\n\nBased on this principle, we conducted experiments to assess the impact of the number of renderings on PointAD's detection performance, circularly rendering point clouds to ensure even coverage of all points. As shown in `Table 5 of the manuscript`, increasing the number of views allows PointAD to gather more detailed information from the 2D renderings, benefiting from smaller angle discrepancies, which improves detection and localization results. However, when the number of views increased from 9 to 11, we observed a performance decline in PointAD, with the I-AUROC for global multimodal detection dropping from 87.4\\% to 86.4\\%. This suggests that incorporating too many views could introduce redundant information, resulting in 2D renderings with extensive overlap and excessive local detail. This overemphasis on local information can impede global recognition.  Hence, the appropriate number of views benefits point understanding from informative views while mitigating the adverse effects of redundant local information.\n\nTo further explore the impact of the absolute angle, we shift the rendering angles while keeping the angle discrepancy unchanged. The original adjacent angle discrepancy in our paper is $\\frac{1}{5}\\pi$. We divide this discrepancy into four parts and perform angle shifts of $\\frac{1}{20}\\pi$, $\\frac{2}{20}\\pi$, and $\\frac{3}{20}\\pi$ to test the impact of varying rendering angles. `Table 6` shows that PointAD maintains consistent performance across different rendering angles, demonstrating its robustness to variations in angles different from those used during training.\n\n**Q2: The robustness under varying lighting and occlusions.**\n\nFurther exploring the robustness of PointAD under different conditions could enhance its generalized detection performance. We conducted ablation studies to test its sensitivity under different lighting conditions and with occluded point clouds below.\n\nTo evaluate the impact of rendering lighting, we adjusted the lighting conditions to render point clouds, generating variant datasets with different lighting. We used both stronger and weaker lighting to render point clouds compared to the original dataset, covering a broad lighting range. We denote stronger and the strongest lighting as \"+\" and \"++\", and weaker and the weakest lighting as \"-\" and \"--\". Visualizations of the resulting samples are presented in `Figure 5`. The experiments were conducted on MVTec3D-AD, where we tested PointAD, trained on the original dataset, on these lighting variant datasets. `Table 7` shows that PointAD can still detect anomalies even with significant discrepancies in rendering lighting, suggesting that PointAD is not sensitive to variations in rendering lighting.\n\nNext, we evaluated detection performance with occluded point clouds. We occluded point clouds by removing those invisible from a specific rendering angle and then used the same rendering parameters to project the remaining points. During this process, we observed that abnormal regions might be occluded totally. Unlike class classification, where class semantics remain unchanged by point occlusions, removing these anomaly semantics would transform anomaly samples into normal points. Therefore, we selected rendering angles that allow visibility of part or all anomalous regions when the point cloud is an abnormal instance. The occluded point clouds are shown in `Figure 6`. We then used PointAD, trained on the original dataset, to test the resulting occluded dataset. `Table 9` shows that PointAD suffers from performance degradation when the point clouds are occluded. We attribute this to two aspects:\n\n1. Despite this strategy, the occluded point clouds could still lose part of the anomaly semantics; \n\n2. Occluded point clouds could create unexpected sinkholes on the surface, which may cause PointAD to identify these areas as hole anomalies incorrectly.\n\n**Q3:  Experiments on additional datasets**\n\nplease refer to the bottom of **Response to Reviewer GgEU** for our response to **Q3** due to character limitation.\n\n**Q4: Discussion of potential limitations and future work.**\n\nWe suggest potential improvements for PointAD as follows: \n\n1. The current rendering process in PointAD does not involve optimization. Using fixed rendering angles for diverse objects might limit detection performance. Adapting rendering angles conditioned on specific objects could make 2D renderings more informative. \n2. PointAD currently treats the 2D renderings as independent instances. Modeling the correlations between multiple 2D renderings based on view angles could enhance both 2D and 3D representations. \n\nIf our responses address your concerns, we would appreciate it if you could consider raising your score to acknowledge our efforts in addressing your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
Dear Reviewers and ACs,

We very much appreciate the insightful and detailed review. We are delighted to hear the encouraging comments from all of the reviewers, including "`novel approach`" (Reviewer **1QVN** and **dthJ**), "`methodological soundness`" (Reviewer **GgEU**), "`stands out for its clarity and easy to follow`" (Reviewer **GgEU** and **dthJ**), and "`robust theoretical foundation and comprehensive empirical validation`" (Reviewer **dthJ**), "`potential for significant impact on future research and practical applications`" (Reviewer **dthJ**), and "`not any obvious weaknesses`" (Reviewer **1QVN**).  While we responded to each of the reviewer comments individually, we also provide a brief summary of the main contents of the rebuttal in response to the reviews:

- In response to the feedback from Reviewer **1QVN**, we have supplemented a deep analysis of the impact of rendering quality, input solution, and point density and distribution.
- As recommended by Reviewer **GgEU**, we have supplemented more baselines to further demonstrate the superiority of our method in performance and computation efficiency.
- To address the concerns raised by Reviewers **dthJ**, we have added more sensitivity analysis (illustrations and experiments) including rendering angles, rendering number, and robustness under different environments. Also, an additional industrial dataset is incorporated into the evaluation.
- We have addressed the feedback from Reviewers **GgEU** by incorporating important references that were previously missing. And, more potential limitations and future work are discussed to respond to Reviewers **dthJ**.

**We put Figures and Tables in the submitted PDF attached to the General Response due to the character limitation.** 

- For Reviewer **1QVN**, we provide visualizations on 2D renderings with different quality in `Figure 1`, low-resolution point clouds in `Figure 2`, and noisy point clouds in `Figure 3`. Also, we provide the corresponding quantitive experimental results in `Tables 1, 2, and 3`.
- For Reviewer **GgEU**, we provide the schematic of multimodality 3D inference in `Figure 4`. `Tables 4 and 5` are supplemented to perform a more comprehensive performance comparison and analysis of computation overhead.
- For Reviewer **dthJ**, `Tables 6, 7, and 8` are added to respond to the concerns about rendering angle, rendering lighting, and point occlusions. Another additional dataset is tested in `Table 9`. `Figures 5 and 6` present the visualizations of point clouds under varying rendering lighting and occluded points. 

For other questions raised by the reviewers, please see our response to individual questions and concerns below each review.

---
# Due to the character limitation, we respond to Q1 for Reviewer GgEU here.

**Q1: Please answer the main differences with these mentioned methods in Weaknesses.**

We thank the reviewer for pointing us toward these concurrent works [1][2]. Since our paper explores an emerging field, i.e., 3D zero-shot anomaly detection, PointAD bridges the gap between zero-shot anomaly detection (previously focused on 2D objects) and 3D anomaly detection. We agree that comparing these approaches could shed light on PointAD. Analysis of the differences between PointAD and these methods are as follows.

**PointAD (3D zero-shot) vs. 2D zero-shot**
First, the scope is different. PointAD transfers the 2D generalization of CLIP to 3D point clouds and has the detection capacity for zero-shot 3D and M3D anomalies. These works aim to capture planar anomalies rather than spatial anomalies. 

- **PointAD vs. (WinCLIP, VAND, and CLIP-AD):** They require extensive human-crafted text prompts to provide representative textual embedding for identifying normality and abnormality. Instead, PointAD can perform zero-shot 3D and M3D anomaly detection using only two learnable generic textual prompts with the help of an auxiliary dataset.

- **PointAD vs. AnomalyCLIP:** AnomalyCLIP limits its prompt learning to the pixel level. In contrast, PointAD proposes hybrid representation learning to capture point and pixel abnormality simultaneously. This mutual collaboration enhances the understanding of both point and pixel anomalies, enabling PointAD to achieve superior performance.

- **PointAD vs. PromptAD:** PromptAD focuses on effectively fusing high-level embeddings to enhance zero-shot detection performance. In contrast, PointAD adapts CLIP from a lower level, prioritizing learning the text prompt to refine the textual embedding.

**PointAD (3D zero-shot) vs. 3D unsupervised**
The differences in settings lead to discrepancies in model design. 3D unsupervised methods fit models to specific categories, limiting their ability to detect unseen objects. In contrast, the zero-shot paradigm learns general decision boundaries, improving the detection of unseen objects.

- **PointAD vs. CLIP3D-AD:** CLIP3D-AD requires extensive human-crafted prompts to produce textual embeddings and learns to capture anomaly semantics only from 2D renderings (pixel anomalies). In contrast, PointAD requires just two learnable text prompts to learn category-agnostic textual embeddings for detecting anomalies in unseen objects. More importantly, PointAD understands 3D anomalies by collaboratively learning from both 3D and 2D perspectives, making it more aware of the spatial relationships of abnormal points.

- **PointAD vs. (IMRNet and M3DM-NR):** They store and model category-wise normal distributions using reconstruction or memory mechanisms. Instead, PointAD leverages CLIP's generalization to model the generic anomaly semantics between textual and visual embeddings. This allows PointAD to achieve both 3D anomaly and multimodal detection.

> **Reference:**
> 1. A survey on visual anomaly detection: Challenge, approach, and prospect.
> 2. Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images.

Meta-review: The submissions got three positive recommendations. The reviewers liked this interesting idea for 3D anomaly detection, but also had some concerns on novelty, limitations, and effectiveness on more datasets. The authors addressed these concerns in the rebuttal and the comments during the authors and reviewers discussion period. The reviewers reached a consensus of acceptance, and did not have an intense discussion during the AC and the reviewers discussion period. The AC read through the paper, review, the confidential comments to the AC, rebuttal, and the discussion. The AC supports the reviewers’ decision, and makes a decision to accept this submission. This decision was approved by the SAC.
Decision: Accept (poster)