Title: PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D anomaly detection

Zero-shot 3D anomaly detection
Abstract: Zero-shot (ZS) 3D anomaly detection is a crucial yet unexplored field that addresses scenarios where target 3D training samples are unavailable due to practical concerns like privacy protection. This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD provides a unified framework to comprehend 3D anomalies from both points and pixels. In this framework, PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. To capture the generic anomaly semantics into PointAD, we propose hybrid representation learning that optimizes the learnable text prompts from 3D and 2D through auxiliary point clouds. The collaboration optimization between point and pixel representations jointly facilitates our model to grasp underlying 3D anomaly patterns, contributing to detecting and segmenting anomalies of unseen diverse 3D objects. Through the alignment of 3D and 2D space, our model can directly integrate RGB information, further enhancing the understanding of 3D anomalies in a plug-and-play manner. Extensive experiments show the superiority of PointAD in ZS 3D anomaly detection across diverse unseen objects.

PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection

Abstract

Zero-shot (ZS) 3D anomaly detection is a crucial yet unexplored field that addresses scenarios where target 3D training samples are unavailable due to practical concerns like privacy protection. This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD provides a unified framework to comprehend 3D anomalies from both points and pixels. In this framework, PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. To capture the generic anomaly semantics into PointAD, we propose hybrid representation learning that optimizes the learnable text prompts from 3D and 2D through auxiliary point clouds. The collaboration optimization between point and pixel representations jointly facilitates our model to grasp underlying 3D anomaly patterns, contributing to detecting and segmenting anomalies of unseen diverse 3D objects. Through the alignment of 3D and 2D space, our model can directly integrate RGB information, further enhancing the understanding of 3D anomalies in a plug-and-play manner. Extensive experiments show the superiority of PointAD in ZS 3D anomaly detection across diverse unseen objects. Code is available at https://github.com/zqhang/PointAD

1 Introduction

Anomaly detection, a significant field within deep learning, has been widely applied to diverse domains, including industrial inspection [2, 3, 32, 36, 37, 44, 15, 24, 55, 6, 19, 63]. While 2D anomaly detection has been extensively studied by exploring RGB information [23, 56, 50, 51, 8, 29], real-world anomalies typically present themselves with abnormal spatial characteristics. Relying solely on RGB information poses challenges in detecting some anomalies in many cases, e.g., when the defect mimics the appearance of the object's background or foreground, as shown in Figure 1(a). The emerging field of 3D anomaly detection aims to unveil these spatial relations indicative of abnormal patterns [22, 4, 45, 9, 54, 13].

However, current 3D anomaly detection methods typically store normal point features during training and identify anomalies by measuring the distance between the test feature and these stored features [22, 54, 13]. They all depend on the assumption that target point clouds are available and entirely normal. This assumption does not hold in various situations when the training samples in the target dataset are inaccessible due to privacy protection ( e.g. , involvement of trade secrets) or the absence of target training data ( e.g. , a new product never seen before) [63]. Figure 1(b) depicts the setting discrepancy between ZS 3D and unsupervised anomaly detection. These methods mentioned above, which detect anomalies by memorizing or reconstructing normal point features, have limitations in generalizing to unseen objects in Figure 1(c). While zero-shot (ZS) anomaly detection has been explored in 2D images [35, 63], ZS 3D anomaly detection remains a research blank. It is a challenging task as ZS

input

Point cloud

input

Ground

truth

RGB

score map

Point

score map

Training

data

Object-

specific Model

Same object

Object-

agnostic Model

Different objects

Test

data

Auxiliary

data

Test

data

Test

data

Test

data

Train

Test

Train

Test

Unsupervised setting

Zero-shot setting

I-AUROC

AUPRO

50

60

70

80

90

100

BTF performance

76.3

92.3

53.7

75.0

Unsupervised

Zero-shot

Figure 1: Motivation of zero-shot 3D anomaly detection. (a) : Top: The hole on the cookies presents a similar appearance to the background. Bottom: Surface damage on the potato is unapparent to the object foreground. In these cases, leveraging RGB information makes it difficult to detect anomalies that imitate the color patterns of the background or foreground. However, effective recognition can be achieved by modeling the point relations within corresponding point clouds. (b) and (c) depicts the setting difference of ZS and unsupervised manner.

3D anomaly detection necessitates the model to detect 3D anomalies across unseen point clouds with diverse class semantics, requiring a robust generalization capacity in the detection model. Recently, Vision-Language Models (VLMs) with their strong generalization capabilities have been applied to various downstream tasks [40, 61, 41, 49, 25, 26]. Particularly, CLIP has demonstrated its strong ZS performance to detect 2D anomalies [35, 24, 63]. Integrating CLIP into the detection model presents a potential solution to the challenging yet unexplored ZS 3D anomaly detection.

In this paper, we propose a unified framework, namely PointAD, to transfer the knowledge of CLIP to detect 3D anomalies in a ZS manner. PointAD comprehends point clouds from both 3D and 2D: (1) deriving 2D representations of point clouds via CLIP by rendering them from multiple views, (2) understanding 3D representations by projecting 2D representations back to 3D, and (3) enhancing 3D comprehension by additional regularization on 2D representations. After grasping point clouds from points and pixels, we propose hybrid representation learning to capture generic normality and abnormality w.r.t. point and pixel information into learnable text prompts [63]. Specifically, since 3D representation manifests its 2D renderings from different views, we treat each representation as one instance and achieve 3D representation aggregation via multiple instance learning (MIL). On this basis, PointAD explicitly aligns the 2D anomalies, rendered from 3D anomalies, to further enhance 3D understanding. We formulate these 2D anomaly recognition tasks from the multi-task learning (MTL) perspective. PointAD collaboratively learns point and pixel representations, promoting the in-depth understanding of underlying abnormal patterns and thus achieving superior ZS normality and abnormality point recognition. Furthermore, benefiting from collaboration optimization, PointAD can directly integrate additional RGB information and perform ZS multimodal 3D (M3D) detection without extra modules and retraining. The main contributions of this paper are summarized as follows:

To the best of our knowledge, we are the first to investigate the challenging yet valuable ZS 3D anomaly detection domain. We propose to transfer the strong recognition generalization of CLIP to detect and segment 3D anomalies over diverse objects.

We introduce a novel ZS 3D anomaly detection approach called PointAD, which provides a unified framework to understand 3D anomalies from points and pixels. Hybrid representation learning is proposed to incorporate the generic normality and abnormality semantics into PointAD, enabling a thorough understanding of 3D anomalies.

PointAD can incorporate 2D RGB information in a plug-and-play manner for testing. In contrast to other methods that require storing RGB information separately, PointAD offers a unified framework to perform ZS M3D anomaly detection directly.

Extensive experiments are conducted to demonstrate the superiority of our model in detecting and segmenting 3D anomalies, even outperforming some unsupervised SOTA methods that memorize normal features of target objects in certain metrics. We hope that our model will serve as a springboard for future research on ZS 3D and M3D anomaly detection.

2 Related Work

3D Anomaly Detection MVTec3D-AD [4], Eyecandies [5], and Real3D-AD [31] provide the point cloud anomalies and the corresponding 2D-RGB information. MVTec3D-AD bridges the connection between 3D and 2D anomaly detection. 3D-ST [4] uses a teacher net to extract dense local geometric

descriptors and design a student net to match such descriptors. AST [45] introduces asymmetric teacher and student net to further improve 3D anomaly detection. IMRNet [28] and 3DSR [58] detect 3D anomalies by reconstruction errors. Instead of only using point clouds, BTF [22], M3DM [54, 53], CPFM [7], and SDM [13] integrate point features and RGB pixel features to detect 3D anomalies. While these approaches exhibit commendable performance by storing object-specific normal point and pixel features within the unsupervised learning framework, such paradigms simultaneously limit their generalization capacity to point clouds from unseen objects, which is crucial to detecting anomalies when the target object is unavailable. To the best of our knowledge, no solution addresses this valuable yet challenging problem. To fill this gap, we introduce PointAD, designed to identify unseen anomalies across diverse objects. PointAD extends CLIP to the realm of ZS 3D anomaly detection and shows robust generalization in capturing generic normality and abnormality within point clouds. Furthermore, PointAD serves as a unified framework, allowing seamless integration of point cloud and RGB modality without additional training.

3D Feature Extraction Conventional methods of 3D feature extraction typically employ a pointbased network like PointNet [38] or PointNet++[39] to extract 3D features from point clouds. Alternative approaches convert 3D data into a 2D format [48, 18], enabling 2D image backbones to process 3D information. PointCLIP [59] directly projects raw points onto image planes for efficiency, but this approach causes the produced depth map to lack geometric details. Instead, rendering-based methods [48, 21] generate 2D renderings by rendering point clouds, allowing for better preservation of local semantics. CPFM [7] stores normal features of these 2D renderings for unsupervised 3D anomaly detection. In this paper, we apply this rendering strategy to the source samples to capture generic anomaly semantics for recognizing abnormalities in unseen objects.

Prompt Learning Instead of fine-tuning the whole network, prompt learning just optimizes the model to adapt the network to downstream tasks. CoOp [61, 60] introduces global context optimization to update learnable text prompts for few-shot recognition. DenseCLIP [41] extends it to the dense classifications. More recently, AnomalyCLIP [63] proposes object-agnostic prompt learning to capture the generic normality and abnormality for images. Our model first introduces hybrid representation learning for ZS 3D anomaly detection, enabling the detection of anomalies and abnormal regions.

3 PointAD

3.1 A Review of CLIP

CLIP, a representative VLM, aligns visual representations to the corresponding textual representations, where an image is classified by comparing the cosine similarity between its visual representation and textual representations of given class-specific text prompts. Specifically, given an image x i and target class set C , visual encoders output the global visual representation f i ∈ R d and local visual representations f i m ∈ R h × w × d , where h , w , and d are the height, width, and dimension, respectively. Textual representations g c are encoded by textual encoder T with the commonly used text prompt template A photo of a [c] , where c ∈ C . The probability of x i belongs to c can be computed as:

where cos ( · , · ) and τ represent the cosine similarity and temperature used in CLIP, respectively. The segmentation S i ( c ) ∈ R h × w for class c can be computed as Seg ( g c , f i m ) , where each entry ( u , v ) is calculated as P ( g c , f m i,u,v ) .

3.2 Overview of PointAD

ZS 3D anomaly detection requires a strong generalization capacity to anomalies on unseen objects with diverse object semantics. In this paper, we propose a unified framework, namely PointAD, to detect and segment 3D anomalies in a ZS manner. In Figure 2, PointAD understands point clouds from both pixel and point perspectives. To make CLIP understand 3D point clouds, we first render point clouds from multiple views and extract the pixel representations of these generated 2D renderings via the visual encoder of CLIP. And then, we derive point representations by projecting these pixel representations back to 3D. Learning generic normality and abnormality is significant in recognizing across-object anomalies. We propose hybrid representative learning, which focuses on glocal point and pixel abnormality, to optimize normality and abnormality text prompts, enabling PointAD with strong generalization to identify 3D anomalies on diverse objects. Benefiting from the hybrid representation learning, PointAD can directly incorporate 2D RGB information during testing to achieve ZS M3D detection.

A A

C

…

…

Multi-view local 2D representations

Multi-view global 2D

representations

Local view-wise 3D representations

R'

-1

A

C

C

C

C

A

R

L3d

local

L3d

global

MTL-based 2D branch

Render(R)

MIL-based 3D branch

For global semantics

For local semantics

…

Vision encoder

Vision encoder

Shared

C

C

C

global

local

C

Lhybrid

Global 3D representations

Learnable

Frozen

Hamiltonian product

Layer-wise aggregation

View-wise Average

A

R'=

ratio

×

R

Cosine similarity

C

C

C

C

gn : normal object-aware/agnostic

learnable text prompts

ga : abnormal object-aware/agnostic

learnable text prompts

Text encoder

…

W1

WE

damaged class

Text encoder

Textual representations

…

W1

WE damaged object

or

…

V1

VE

class

…

V1

VE

object

or

1

0.95

1

0.92

口

个

个

3.3 Multi-View Rendering

Multi-view projection is a crucial technology for understanding point clouds from 2D perspectives. Some multi-view projection approaches project point clouds into various depth maps, providing adequate shape information for class recognition [59]. However, in this paper, our objective is to learn both generic global and local anomaly semantics. Depth-map projection lacks sufficient resolution to represent fine-grained anomaly semantics accurately. Hence, we adopt high-precision rendering to preserve the original 3D information offline. Specifically, given an auxiliary dataset of point clouds D 3 d = { ( x 3 d i , y 3 d i ) } N i =1 , we define the rendering matrix as R ( k ) for the k -th view, with a total of K views. We simultaneously render point clouds and point-level ground truths from different views to obtain their corresponding 2D renderings, which is given by x ( k ) i = R ( k ) ( x 3 d i ) and y ( k ) i = R ( k ) ( y 3 d i ) , where x ( k ) i ∈ R H × W and y ( k ) i ∈ R H × W respectively represent the k -th 2D renderings and corresponding pixel-level ground truth in the i -th point cloud. Note that anomaly pixels are marked as 1, and normal pixels are marked as 0.

3.4 Representations for 3D and 2D information

PointAD aims to learn generic anomaly semantics from both 3D and 2D representations, enabling a comprehensive understanding of point and pixel anomaly patterns. For a point cloud x 3 d i , we first obtain the 2D renderings X i = { x ( k ) i } K k =1 . Then, these renderings are encoded via the vision encoder of CLIP to obtain global 2D representations F i = { f ( k ) i } K k =1 , and local 2D representations F i m = { f m ( k ) i } K k =1 . As for point cloud representations, we consider that one point cloud will be projected into multiple 2D renderings. Consequently, global 3D representation p i and local 3D representations p i m are expected to include their corresponding 2D representations in each view. Formally, p i = { p ( k ) i | p ( k ) i = f ( k ) i } K k =1 and p i m = { p m ( k ) i | p m ( k ) i = { p m ( k ) i,j } n j =1 } K k =1 , where p m ( k ) i,j = { p m ( k ) i,j = f m ( k ) i,u,v , ( u, v ) = R ′ ( k ) ( a i,j | b i,j , c i,j ) } represents the j -th point representation of i -th point cloud in the k -th view, whose 3D coordinate is ( a i,j , b i,j , c i,j ). R ′ ( k ) is the rendering transformation between the point and pixel representation, derived as R ′ ( k ) = h H R ( k ) .

Points at different positions may yield a different number of 2D representations as they are hidden by other points from a specific viewpoint. In this case, we introduce a view-wise visibility mask M , where M k i,j indicates whether the j -th point of the i -th point cloud is visible in the k -th view. We compare the depth of points projected into the same pixel in the same view and set the corresponding visibility mask to 1 for the point with the minimum depth, and to 0 for the other points. Let Q ( k ) i,u,v denote the depths set of all points that are projected into the same pixel indexed by ( u, v ) in the i -th point cloud in the k -th view. Q ( k ) i,u,v and M k i,j are respectively given as Q ( k ) i,u,v = { c i,j | R ′ ( k ) ( a i,j , b i,j , c i,j ) = ( u, v ) } n j =1 and M ( k ) i,j = I ( i, j, k = argmin i,j,k { c i,j | c i,j ∈ Q ( k ) i,u,v } ) , where I ( · ) is an indicator function. Local 3D representations of the i -th point cloud for the k -th view are reformulated as: p m ( k ) i = { p m ( k ) i,j ∗ M ( k ) i,j } n j =1 .

3.5 Hybrid representation learning

The key of ZS 3D anomaly detection requires the model to capture generic anomaly semantics, rather than relying on specific object semantics. Since CLIP was originally pre-trained to align object semantics, such alignment harms the generalization capacity of CLIP to recognize anomalies on various objects. To adapt CLIP to 3D anomaly detection, we propose a hybrid representation learning, from both 3D and 2D perspectives, to globally and locally optimize textual representations. This enables PointAD to learn more representative text embedding for glocal anomaly semantics alignment. Following previous work [63, 61], we randomly initialize two learnable text templates t n and t a , in AnomalyCLIP [63] or CoOp manner [61], to obtain more overall text embeddings g n and g a to recognize normality and abnormality, respectively.

where V and W are learnable word embeddings, respectively.

MIL-based 3D representation learning To fully incorporate 3D glocal anomaly semantics into PointAD from point information, we respectively devised two losses to capture 3D global anomalies and local anomaly regionals. First, we compute the cosine similarity between the textual representation and its rendering global representations in each view. As point clouds are projected from different views, the resulting renderings in each view reflect certain parts of point clouds. We use view-wise MIL to integrate 2D global representations and then align global labels to capture the global semantics. Formally, the global 3D loss is defined as:

As for local point anomaly semantics, we quantify the cosine similarity between textual representations and local representations of 2D renderings. Since points within point clouds are projected from different views, their projections in each view present part characteristics of themselves. We adopt the pixel-wise MIL to achieve the aggregation of point local representation. The point segmentation can be formulated mathematically as follows:

However, deriving such 3D segmentation requires similarity computation for each point. It brings a significant memory burden, with a huge computational complexity of O ( Knd ) , which is unaffordable for one NVIDIA RTX 3090 24GB GPU. To address this computational challenge, we resort to the rendering correspondence between points (3D space) and their corresponding pixels within each view (2D space). We first can rewrite 3D segmentation from the view perspective as S 3 d i ( a ) = 1 K ∑ k S 3 d ( k ) i ( a ) . Then, the k -th division of 3D segmentation can be transformed into the 2D counterpart through the rendering projection S 3 d ( k ) i ( a ) = ( R ( k ) ) ( -1) S 2 d ( k ) i ( a ) ⊗ M ( k ) i , where ⊗ is the Hamiltonian product. The k -th 2D counterpart can be computed as S 2 d ( k ) i ( a ) = Up ( Seg ( g a , f m ( k ) i )) , where the operator Up ( · ) represents bilinear interpolation from feature space to 2D space. Finally, we can reformulate the 3D

segmentation as follows:

From the equation, we can observe that the primary computation can be conducted in the feature space, with a computational complexity of O ( Khwd ) . This is a substantial overhead reduction compared to O ( Knd ) since feature space is much smaller than 3D space, i.e. , h × w ≪ n . In our experiment, h × w = 24 × 24 = 576 , while n = 336 × 336 = 112896 . With this transformation, the

entire experiment can be conducted using only a single NVIDIA RTX 3090 24GB GPU. After that, Dice Loss is employed to precisely model the decision boundary of anomaly regions. Let I represent a full-one matrix of the same size as y 3 d i . Formally, we define 3D local loss L local 3 d :

MTL-based 2D representation learning We further improve PointAD point understanding by capturing 2D glocal anomaly semantics into the object-agnostic text prompt template. We treat the anomaly recognition for one rendering from the point cloud as a task. Hence, we formulate the anomaly semantics learning for multiple 2D renderings as MTL. MTL-based 2D representation learning is divided into two parts for respective alignment to 2D global and local anomaly semantics. For 2D global semantics, we use CrossEntropy to quantify the discrepancy between the textual representations and each global 2D representation. Global MTL-based 2D representation learning L global 2 d is defined as:

Also, we focus on 2D abnormal regions to understand pixel-level anomalies. As the anomaly regions are typically smaller than normal regions, we employ Focal Loss to mitigate the class imbalance besides Dice Loss. Let ⊕ denote the concatenation operation. Local MTL-based 2D representation learning L local 2 d is given as follows:

3.6 Training and Inference

PointAD detects 3D anomalies from both 3D and 2D perspectives and thus combing these above losses to derive hybrid loss L hybrid . We minimize L hybrid to incorporate generic anomaly semantics into the text prompt from point and pixel spaces:

During training, we minimize the hybrid loss L hybrid , where the original parameters of CLIP are frozen to maintain its strong generalization. Since our model provides a unified framework to understand anomaly semantics from point and pixel, it can not only perform ZS 3D anomaly detection but also M3D anomaly detection in a plug-and-play way . Next, we will introduce the inference process in detail:

ZS 3D/M3D inference Given a point cloud x 3 d i , we regard the 3D segmentation (See Equ. 2) as the anomaly score map: A i m = G σ ( S 3 d i ( a ) ), where G σ ( · ) represents the Gaussian filter. The global anomaly score incorporates glocal anomaly semantics and is computed as A s i = 1 2 ( 1 K ∑ f ( k ) i ∈F i P ( g c , f ( k ) i )+ max( A i m )) . When the RGB counterpart is available for testing, PointAD could directly integrate RGB information by feeding RGB images to 2D branch to derive 2D representations. We project these 2D representations back to 3D branch to respectively compute RGB anomaly score map and anomaly score as A m ( rgb ) i = P ( g c , f ( rgb ) i ) and A s ( rgb ) i = G σ ( S 3 d ( rgb ) i ( a ) ) . The final multimodal anomaly score map and anomaly score are defined as A m ( mod ) i = 1 2 G σ ( A i m + A m ( rgb ) i ) and A s ( mod ) i = 1 2 [ 1 2 ( A s ( rgb ) i + A s i ) + max ( A m ( mod ) i ) ] , respectively.

4 Experiment

4.1 Experiment Setup

Dataset Weevaluate the performance of ZS 3D anomaly detection on three public datasets including, MVTec3D-AD, Eyecandies and Real3D-AD. MVTec3D-AD, Eyecandies, and Real3D-AD are multiclass datasets and respectively contain 10 classes, 10 classes, and 12 classes. Since these training datasets only contain all normal samples, we use the common zero-shot setting one-vs-rest, where an object test dataset is used to fine-tune PointAD and assess the ZS anomaly detection for the remaining objects. We also explore a more challenging setting: cross-dataset ZS generalization, which requires the detection model to generalize to anomalies on other datasets. For point cloud anomaly detection, we only use point clouds to detect and localize 3D anomalies. In M3D anomaly detection, the 2D RGB information is utilized only for testing. To comprehensively analyze PointAD, we utilize four metrics to assess its performance in both anomaly detection and segmentation.

Input

View 1

score map

View 2

score map

View 3

score map

View 4

score map

View 5

score map

View 6

score map

View 7

score map

View 8

score map

View 9

score map

Point

score map

Ground

truth

4.2 Implementation Details & Baselines

Both point clouds and 2D renderings are resized to 336 × 336. We use Open3d library to generate 9 views by rotating point clouds along the X-axis at angles of {-4 5 π, -3 5 π, -2 5 π, -1 5 π, 0 , 1 5 π, 2 5 π, 3 5 π, 4 5 π } for most categories. We circularly set the rendering angles, evenly distributing the angles between -π to π . The backbone of PointAD is the pre-trained CLIP model ( VIT-L/14@336px in open_clip ). Following [63], we improve the local visual semantics of vision encoder of CLIP without modifying its parameters. During training, we keep all parameters of CLIP frozen and set the learnable word embeddings in object-agnostic text templates to 12 . All experiments were conducted on a single NVIDIA RTX 3090 24GB GPU using PyTorch-2.0.0. As there is no work to explore the field of ZS 3D anomaly detection, we make a great effort to provide these comparisons. We apply the original CLIP to our framework for 3D detection, called CLIP + Rendering. Also, we reproduce SOTA 3D recognition works including PointCLIP V2 [64] and Cheraghian [12], and adapt them for ZS 3D anomaly detection. We compare the SOTA 2D anomaly detection approach AnomalyCLIP [63] by fine-tuning it on depth maps. PointAD by default uses object-agnostic text prompts, whereas PointAD-CoOp employs object-aware prompts. Appendix B and C provide more details on implementation and baselines.

4.3 Main Results

We fine-tuned PointAD on three objects on MVTec3D-AD, Eyecandies, and Real3D-AD. Over three runs, the averaged results on one-vs-rest and cross-dataset settings are reported. We use the metric pairs (I-AUROC% % ↑ and AP% % ↑ ) and (P-AUROC% % ↑ and AUPRO% % ↑ ) to evaluate the glocal detection performance, respectively. Details of experimental settings see Appendix A. The best and second-best results in ZS are highlighted in Red and Blue. G. and L. represent 3D global and local anomaly detection. M3D global and local anomaly detection are abbreviated as MG. and ML.

ZS 3D anomaly detection Table 1 presents the comparison of ZS 3D performance. Compared to the point-based method Cheraghian and the projection-based method PointCLIP V2, PointAD achieves superior performance on ZS 3D anomaly detection over all three datasets. Especially, it outperforms CLIP + Rendering from 61.2% to 82.0% I-AUROC and from 85.8% to 94.2% AP on MVTec3D-AD. In addition, PointAD achieves superior segmentation performance on ZS 3D anomaly detection, improving MVTec3D-AD by a large margin compared to Cheraghian from 88.2%

2D RGB

input

Point cloud

input

Ground

truth

RGB

score map

Point

score map

Multimodal

score map

(a) Multimodal visualization with hybrid loss.

2D RGB

input

Point cloud

input

Ground

truth

RGB

score map

Point

score map

Multimodal

score map

(b) Multimodal visualization without 2D glocal loss.

Figure 4: Visualization comparison between PointAD with hybrid loss and without.

to 95.5% P-AUROC and from 57.0% to 84.4% AUPRO. This improvement in overall performance is attributed to PointAD adapting CLIP's strong generalization to glocal anomaly semantics through hybrid representation learning. In addition, PointAD advances PointAD-CoOp across all datasets by blocking the class semantics in text prompts [63].

ZS M3D anomaly detection We also compare the ZS M3D anomaly detection when RGB information is available for testing. As shown in Table 2, the results indicate that PointAD can integrate additional RGB information and further boost its performance from 82.0% to 86.9% AUROC and from 94.2% to 96.1% AP for global semantics on MVTec3D-AD. Additionally, as for local semantics, the performance improves from 95.5% to 97.2% P-AUROC and from 84.4% to 90.2% AUPRO. A large performance gain is also obtained on Eyecandies and Real3D-AD. While other methods improve their performance in some metrics, they still suffer from performance degradation in other metrics due to inefficient integration of the two modalities. Instead, PointAD achieves overall improvement across all metrics by incorporating explicit joint constraints on both point and pixel information.

Cross-dataset ZS anomaly detection We perform the cross-dataset anomaly recognition to further evaluate the zero-shot capacity of PointAD, where we use one object as the auxiliary and test objects with totally different semantics and scenes in another dataset. We compare all baselines that need fine-tuning. From Table 3 and M3D from

Table 4, PointAD demonstrates strong cross-dataset generalization performance on Eyecandies and Real3D-AD, with nearly no obvious performance decay compared to the one-vs-rest setting. The strong transfer ability highlights its robust generalization capabilities in detecting anomalies in objects with unseen semantics and backgrounds.

4.4 Result Analysis

Visualization analysis. To intuitively present the strong generalization capacity of our model to unseen anomalies, we visualize the anomaly score maps of the 3D and corresponding 2D counterparts of PointAD on MVTec3D-AD. As shown in Figure 3, PointAD reveals abnormal spatial relationships of points and further captures the generic point anomaly patterns across diverse objects. And, we also visualize the anomaly score map of corresponding 2D counterparts, where 3D point anomalies are transformed into 2D pixel anomalies. It can be observed that PointAD also has a strong detection ability on such 2D anomalies. The strong representative pixel representations from multiple views facilitate more precise 3D anomaly detection. Quantitative results are provided in Section 5. The strong 3D and 2D detection capabilities of PointAD are from hybrid representation learning, which not only enables PointAD to capture the 3D anomalies but also explicitly constrains 2D representations.

How multimodality makes PointAD accurate. PointAD is a unified framework that can not only capture point anomalies but also handle 2D information in a plug-and-play manner. As shown in Figure 4(a), we visualize M3D results of PointAD on MVTec3D-AD. The surface damage on the potato presents a similar appearance to the object foreground, which makes it difficult to detect this anomaly with RGB information. On the contrary, the point relations for the color stain on foam are the same as those of normal, but they have a clear distinction in the RGB information. PointAD

can integrate these two modalities, thereby complementing their respective advantages. We further investigate the reason why PointAD can directly leverage both modalities. For this purpose, we experiment without 2D glocal loss. As shown in Figure 4(b), without 2D glocal loss, significant noise disrupts and even covers the RGB score maps, resulting in unpromising multimodal fusions. This illustrates the importance of explicit constraints on the 2D space. Hence, we conclude that the robust multimodal detection capability of our model stems from the collaboration optimization in both 3D and 2D spaces during training. We provide more analysis about failure cases and the computation overhead in Appendix G and H.

5 Ablation Study

Module Ablation. Here, we investigate the effectiveness of the proposed main technologies by progressively adding the proposed modules. Table 6 illustrates that vanilla , which represents the aforementioned CLIP + rendering, performs poor results on both 3D and M3D anomaly detection because CLIP focuses on alignment for 2D object semantics instead of anomaly semantics. With the 3D global branch, we incorporate the global anomaly semantics into PointAD, improving overall performance in local and global detection. After adding the 3D local branch, the performance is further improved, while the pixel-level performance on M3D detection suffers from performance degradation. This is attributed to the absence of 2D constraints, leading to inefficient multimodality fusion as we integrate 2D RGB information in a plug-and-play way. The inclusion of 2D global branch explicitly incorporates 2D anomaly information, which makes PointAD obtain overall performance gain. Finally, by further focusing on 2D anomaly regions, PointAD has a deeper understanding of point clouds from 2D representations and promotes multimodality fusion. Therefore, our model notably boosts the multimodal segmentation performance from 92.9% to 97.2% P-AUROC and from 84.4% to 90.2% AUPRO.

View Number Ablation. PointAD interprets point clouds from 2D renderings, and the quantity of rendering views directly affects the 3D original information acquired by PointAD. Table 5 depicts that the appropriate number of views benefits point understanding from informative views while alleviating negative effects of subpar views. More ablations about the length of learnable prompts, layers of intermediate vision features, and the number of training sets are provided in Appendix F.

6 Conclusion

This paper takes the first attempt to study the challenging yet underexplored tasks of ZS 3D and M3D anomaly detection. We propose a unified framework, namely PointAD, to transfer the strong generalization of CLIP to 3D point clouds. PointAD understands 3D anomalies from 3D and 2D spaces. Benefiting from hybrid representation learning, PointAD can recognize generic 3D normality and abnormality across diverse objects and directly integrate RGB information for ZS M3D. Extensive experiments demonstrate the superior ZS detection capacity of our model, whether single modality or multimodality. Code will be made available once the paper is accepted.

Limitations PointAD utilizes fixed rendering angles to generate 2D renderings across diverse objects. While experimental results demonstrate its superiority, the development of a fine-grained filtering mechanism to select high-quality 2D renderings, particularly for revealing anomalies, remains an avenue for future research.

Broader Impact Our paper aims to enhance automated detection and decision-making in smart manufacturing, which does not involve any potential ethical risks. Since the collection of 3D samples is more labor-intensive and costly, our research on using vision-language models for zero-shot point cloud detection can have significant societal impacts, especially in scenarios where target 3D training samples are unavailable due to privacy concerns or the absence of products. We hope that our first exploration of the ZS 3D anomaly detection field could pave the way for further research in this emerging field.

Acknowledgments

This work was supported by NSFC U23A20326 and NSFC 62088101 Autonomous Intelligent Unmanned Systems.

References

Toshimichi Aota, Lloyd Teh Tzer Tong, and Takayuki Okatani. Zero-shot versus many-shot: Unsupervised texture anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5564-5572, 2023.

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9592-9600, 2019.

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4183-4192, 2020.

Paul Bergmann and David Sattlegger. Anomaly detection in 3d point clouds using deep geometric descriptors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2613-2623, 2023.

Luca Bonfiglioli, Marco Toschi, Davide Silvestri, Nicola Fioraio, and Daniele De Gregorio. The eyecandies dataset for unsupervised multimodal anomaly detection and localization. In Proceedings of the Asian Conference on Computer Vision (ACCV) , pages 3586-3602, December 2022.

Tri Cao, Jiawen Zhu, and Guansong Pang. Anomaly detection under distribution shift. arXiv preprint arXiv:2303.13845 , 2023.

Yunkang Cao, Xiaohao Xu, and Weiming Shen. Complementary pseudo multimodal feature for point cloud anomaly detection. arXiv preprint arXiv:2303.13194 , 2023.

Yunkang Cao, Xiaohao Xu, Jiangning Zhang, Yuqi Cheng, Xiaonan Huang, Guansong Pang, and Weiming Shen. A survey on visual anomaly detection: Challenge, approach, and prospect. ArXiv , abs/2401.16402, 2024.

Ruitao Chen, Guoyang Xie, Jiaqi Liu, Jinbao Wang, Ziqi Luo, Jinfan Wang, and Feng Zheng. Easynet: An easy network for 3d industrial anomaly detection. In Proceedings of the 31st ACM International Conference on Multimedia , pages 7038-7046, 2023.

Xuhai Chen, Yue Han, and Jiangning Zhang. A zero-/few-shot anomaly classification and segmentation method for cvpr 2023 vand workshop challenge tracks 1&2: 1st place on zero-shot ad and 4th place on few-shot ad. arXiv preprint arXiv:2305.17382 , 2023.

Yuanhong Chen, Yu Tian, Guansong Pang, and Gustavo Carneiro. Deep one-class classification via interpolated gaussian descriptor. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 383-392, 2022.

Ali Cheraghian, Shafin Rahman, Townim F Chowdhury, Dylan Campbell, and Lars Petersson. Zero-shot learning on 3d point cloud objects and beyond. International Journal of Computer Vision , 130(10):2364-2384, 2022.

Yu-Min Chu, Chieh Liu, Ting-I Hsieh, Hwann-Tzong Chen, and Tyng-Luh Liu. Shape-guided dual-memory learning for 3d anomaly detection. In International Conference on Machine Learning , pages 6185-6194. PMLR, 2023.

Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357 , 2020.

Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9737-9746, 2022.

Hanqiu Deng, Zhaoxiang Zhang, Jinan Bao, and Xingyu Li. Anovl: Adapting vision-language models for unified zero-shot anomaly localization. arXiv preprint arXiv:2308.15939 , 2023.

Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 6568-6576, 2022.

Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In International Conference on Machine Learning , pages 3809-3820. PMLR, 2021.

Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting industrial anomalies using large vision-language models, 2023.

Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 98-107, 2022.

Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1-11, 2021.

Eliahu Horwitz and Yedid Hoshen. Back to the feature: classical 3d features are (almost) all you need for 3d anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2967-2976, 2023.

Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang. Registration based few-shot anomaly detection. In European Conference on Computer Vision , pages 303-319. Springer, 2022.

Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winclip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19606-19616, 2023.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113-19122, 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Zero-shot anomaly detection via batch normalization. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Wenqiao Li, Xiaohao Xu, Yao Gu, Bozhong Zheng, Shenghua Gao, and Yingna Wu. Towards scalable 3d anomaly detection and localization: A benchmark via 3d anomaly synthesis and a self-supervised learning network. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 22207-22216, 2023.

Xurui Li, Ziming Huang, Feng Xue, and Yu Zhou. Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images. ArXiv , abs/2401.16753, 2024.

Yiting Li, Adam David Goodge, Fayao Liu, and Chuan-Sheng Foo. Promptad: Zero-shot anomaly detection using text prompts. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1082-1091, 2024.

Jiaqi Liu, Guoyang Xie, ruitao chen, Xinpeng Li, Jinbao Wang, Yong Liu, Chengjie Wang, and Feng Zheng. Real3d-AD: A dataset of point cloud anomaly detection. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller. Explainable deep one-class classification. arXiv preprint arXiv:2007.01760 , 2020.

Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus-Robert Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint arXiv:2205.11474 , 2022.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-ofdistribution detection with vision-language representations. Advances in Neural Information Processing Systems , 35:35087-35102, 2022.

Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van den Hengel. Explainable deep few-shot anomaly detection with deviation networks. arXiv preprint arXiv:2108.00462 , 2021.

Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR) , 54(2):1-38, 2021.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18082-18091, 2022.

Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2806-2814, 2021.

Tal Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2155-2162, 2023.

Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14318-14328, 2022.

Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Asymmetric studentteacher networks for industrial anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2592-2602, 2023.

Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning , pages 4393-4402. PMLR, 2018.

Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John Shawe-Taylor, John C Platt, et al. Support vector method for novelty detection. In NIPS , volume 12, pages 582-588. Citeseer, 1999.

Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision , pages 945-953, 2015.

Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems , 35:30569-30582, 2022.

Yiyou Sun, Zhenmei Shi, Yingyu Liang, and Yixuan Li. When and how does known class help discover unknown ones? provable understandings through spectral analysis. In International Conference on Machine Learning , 2023.

Yu Tian, Fengbei Liu, Guansong Pang, Yuanhong Chen, Yuyuan Liu, Johan W Verjans, Rajvinder Singh, and Gustavo Carneiro. Self-supervised pseudo multi-class pre-training for unsupervised anomaly detection and segmentation in medical images. Medical Image Analysis , page 102930, 2023.

Yu Tian, Guansong Pang, Fengbei Liu, Yuanhong Chen, Seon Ho Shin, Johan W Verjans, Rajvinder Singh, and Gustavo Carneiro. Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in medical images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part V 24 , pages 128-140. Springer, 2021.

Chengjie Wang, Haokun Zhu, Jinlong Peng, Yue Wang, Ran Yi, Yunsheng Wu, Lizhuang Ma, and Jiangning Zhang. M3dm-nr: Rgb-3d noisy-resistant industrial anomaly detection via multimodal denoising. arXiv preprint arXiv:2406.02263 , 2024.

Yue Wang, Jinlong Peng, Jiangning Zhang, Ran Yi, Yabiao Wang, and Chengjie Wang. Multimodal industrial anomaly detection via hybrid fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8032-8041, 2023.

Guoyang Xie, Jingbao Wang, Jiaqi Liu, Feng Zheng, and Yaochu Jin. Pushing the limits of fewshot anomaly detection in industry vision: Graphcore. arXiv preprint arXiv:2301.12082 , 2023.

Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. Advances in Neural Information Processing Systems , 35:4571-4584, 2022.

Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows. arXiv preprint arXiv:2111.07677 , 2021.

Vitjan Zavrtanik, Matej Kristan, and Danijel Skoˇ caj. Cheating depth: Enhancing 3d surface anomaly detection via depth simulation. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2153-2161, 2023.

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8552-8562, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Qihang Zhou, Shibo He, Haoyu Liu, Tao Chen, and Jiming Chen. Pull & push: Leveraging differential knowledge distillation for efficient unsupervised anomaly detection and localization. IEEE Transactions on Circuits and Systems for Video Technology , 2022.

Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Objectagnostic prompt learning for zero-shot anomaly detection, 2023.

Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2639-2650, 2023.

A Dataset

Dataset We evaluate the performance of ZS 3D anomaly detection on three publicly available 3D anomaly detection datasets, MVTec3D-AD, Eyecandies, and Real3D-AD. MVTec3D-AD comprises 4147 point clouds across 10 categories. These objects exhibit diverse object semantics, including bagel, cable gland, carrot, cookie, dowel, foam, peach, potato, rope, and tire. The training dataset comprises 2656 normal point clouds, and the validation dataset comprises 294 normal point clouds. The test dataset includes 948 normal and 249 anomaly point clouds, covering several anomaly types. Point-wise annotations are available for the point clouds. MVTec3D-AD also provides corresponding 2D-RGB image counterparts for the point clouds. We remove the background plane of point clouds in the whole dataset like [22]. Eyecandies also has 10 different classes and provides the corresponding 2D RGB information. Real3D-AD is a recently available dataset, which contains 12 objects. However, it does not provide the RGB information.

Evaluation Setting and Metric Since the training dataset of MVTec3D-AD only contains all normal samples, we use an object test dataset as the auxiliary dataset to fine-tune PointAD and assess the ZS anomaly detection for the remaining objects. In particular, we report the average results using different objects as the auxiliary, i.e., carrot, cookie, and dowel for MVTec3DAD; confetto, LicoriceSandwich, and PeppermintCandy for Eyecandie; seahorse, shell, and starfish for Real3D-AD. Moreover, we explore more challenging cross-dataset generalization settings, where we use auxiliary data to test all objects of another dataset. For point cloud anomaly detection, we only use point clouds to detect and localize 3D anomalies in Figure 5(a). In M3D anomaly detection, both point clouds and their 2D-RGB counterparts are utilized, as shown in Figure 5(b). To comprehensively analyze PointAD, we utilize four metrics to assess its anomaly classification and segmentation performance. For anomaly detection, we use the Area Under the Receiver Operating Characteristic Curve (I-AUROC % ↑ ) and average precision (AP % ↑ ). Regarding anomaly segmentation, we use point-level AUROC (P-AUROC % ↑ ) and a restricted metric called AUPRO %( ↑ ) [3] to provide a detailed evaluation of subtle anomaly regions.

Zero-shot 3D anomaly detection

R

2D branch

R'

-1

3D branch

Anomaly score

Anomaly map

Text

prompts

Point clouds

R

2D branch

R'

-1

3D branch

Anomaly score

Anomaly map

Zero-shot multimodal anomaly detection

RGB counterpart

Text

prompts

Point clouds

(a) Inference for ZS 3D anomaly detection

(b) Inference for ZS M3D anomaly detection

B Implementation Details

Both point clouds and 2D renderings are resized to 336 × 336. We use Open3d library 2 to generate 9 views by rotating point clouds along the X-axis at angles of

{-4 5 π, -3 5 π, -2 5 π, -1 5 π, 0 , 1 5 π, 2 5 π, 3 5 π, 4 5 π } for most categories. Some categories lose their surface completely because they are not stereo point clouds. Table 7 also gives the specific rendering angles of different view number settings. We set the rendering angles in a circular manner, evenly distributing the angles between -π to π . The backbone of PointAD is the pre-trained CLIP model 3 ( VIT-L/14@336px ). Following [63], we improve the local visual semantics of the vision encoder of CLIP without modifying its parameters and introduce learnable tokens in the text encoder. During training, we keep all CLIP parameters frozen and set the learnable word embeddings in objectagnostic text templates to 12 . We use the Adam optimizer with a learning rate of 0 . 001 to optimize the learnable parameters. The experiment runs for 15 epochs with a batch size of 4 . All experiments were conducted on a single NVIDIA RTX 3090 24GB GPU using PyTorch-2.0.0.

C Baselines

ZS 3D anomaly detection and M3D anomaly detection have not yet been explored. We first make an adaption for the original CLIP for 3D anomaly detection. Then, we reproduce SOTA ZS 3D classification methods ( i.e. , PointCLIP V2 [64] and Cheraghian [12]) and adapt them to our settings. SOTA unsupervised 3D anomaly detection approaches are reported as the performance upper bound. All hyperparameters in these baselines are kept the same. We will present the detailed reproduction as follows:

CLIP + Rendering is a method, where we apply the original CLIP into our framework for ZS 3D anomaly detection. It uses the same rendering procedure as PointAD. Following [24, 63], we integrate anomaly semantics into CLIP by two class text prompt templates: A photo of a normal [cls] and A photo of an anomalous [cls] , where cls denotes the target class name.

PointCLIP V2 (CVPR 2023) is a SOTA ZS 3D classification method based on CLIP, they project point clouds into depth maps from different views. To adapt PointCLIP V2 into ZS anomaly detection, we replace its original text prompts point cloud of a big [c] with normal text prompts point cloud of a big [c] and abnormal text prompts point cloud of a big damaged [c] .

AnomalyCLIP (ICLR 2024) is a SOTA zero-shot 2D anomaly detection method. AnomalyCLIP introduces object-agnostic learning to capture generic anomaly semantics of images. We adapt AnomalyCLIP in 3D detection by fine-tuning AnomalyCLIP on depth images of corresponding point clouds.

Cheraghian (IJCV 2022) is an approach for ZS 3D classification without foundation models. They directly extract the point presentations by PointNet and use word2vector [34] to generate the textual embedding of an object. To incorporate the anomaly semantics into Cheraghian, we average the textual embeddings of [c] and damaged . We replace the global representation with dense representations to provide the segmentation to provide the segmentation results.

D Related Work

D.1 2D Anomaly Detection

2D anomaly detection has been studied extensively by leveraging RGB information [47, 46, 14, 52, 42, 11, 62, 43]. Related works can be categorized into two branches: end-to-end and memorybased methods. Representative end-to-end methods exploit knowledge distillation [3, 62, 45] and normalizing flow [20, 57] to model the normal distribution. Instead, memory-based methods [44, 55] store normal features to construct normal prototypes. ZS 2D anomaly detection is proposed to target a challenging problem where training samples are inaccesible [17, 27, 33, 1, 16, 10]. WinCLIP [24] attempts to explore ZS 2D anomaly detection using CLIP. AnomalyCLIP [63] first introduces objectagnostic prompt learning to capture generic normality and abnormality, detecting anomalies across datasets. PromptAD [30] focuses on effectively fusing these embeddings to enhance zero-shot detection performance.

Sigma 1

Sigma 5

Original

Sigma 9

Ground Truth

20%

Original

30%

50%

70%

Ground Truth

E Analysis on Rendering Conditions

Rendering quality PointAD interprets point clouds through their corresponding 2D renderings, and the quality of these renderings impacts the information that PointAD can extract from the original point clouds. In our manuscript, we used the Open3D Library to render the point clouds, but it does not provide an API for controlling rendering quality. To simulate varying rendering quality, we applied Gaussian blur with different extents σ to the 2D renderings. Sample visualizations are included in Figure 6. Specifically, we conducted experiments on MVTec3D-AD using different blur σ values (i.e., ). Table 8 shows that the detection performance of PointAD diminishes as rendering quality decreases (with increasing sigma). However, the degradation is acceptable even when the renderings are heavily blurred ( σ equals 9). In such cases, PointAD still outperforms baselines that use high-quality renderings.

Input Resolution Here, we study the effect of resolutions of input point clouds. To create lowresolution point clouds, we downsample the entire high-resolution point clouds using Farthest Point Sampling (FPS) with various sampling ratios. This strategy allows us to generate corresponding low-resolution datasets for training and evaluating PointAD. Visualizations of these datasets are provided in Figure 7. We train PointAD using the resulting low-resolution samples and test PointAD on the same resolution. Table 9 demonstrates that PointAD maintains a strong detection capacity for low-resolution point clouds when the downsampling ratios are 20%, 30%, 50%, and 70%. Even at 20% resolution, PointAD still achieves state-of-the-art performance. This indicates that PointAD is generally applicable to point clouds with various resolutions.

Rendering angles and different numbers of views PointAD interprets point clouds from their 2D renderings, where rendering angles and numbers collectively determine the amount of information derived. They have different emphases. For rendering angles, the importance lies in the discrepancy between adjacent angles, as this affects the information granularity that PointAD obtains from adjacent views. When the angle discrepancy is fixed, the number of renderings determines the coverage of 3D information in the resulting 2D renderings. To capture all point cloud information, especially abnormal points, it is crucial to ensure comprehensive coverage. Therefore, our approach in selecting rendering angles and the number of renderings is to guarantee that all points in point clouds are adequately represented.

Based on this principle, we conducted experiments to assess the impact of the number of renderings on PointAD's detection performance, circularly rendering point clouds to ensure even coverage of all points. As shown in Table 5, increasing the number of views allows PointAD to gather more detailed information from the 2D renderings, benefiting from smaller angle discrepancies, which improves detection and localization results. However, when the number of views increased from 9 to 11, we observed a performance decline in PointAD, with the I-AUROC for global multimodal detection dropping from 87.4% to 86.4%. This suggests that incorporating too many views could introduce redundant information, resulting in 2D renderings with extensive overlap and excessive

local detail. This overemphasis on local information can impede global recognition. Hence, the appropriate number of views benefits point understanding from informative views while mitigating the adverse effects of redundant local information. To further explore the impact of the absolute angle, we shift the rendering angles while keeping the angle discrepancy unchanged. The original adjacent angle discrepancy in our paper is 1 5 π . We divide this discrepancy into four parts and perform angle shifts of 1 20 π , 2 20 π , and 3 20 π to test the impact of varying rendering angles. Table 10 shows that PointAD maintains consistent performance across different rendering angles, demonstrating its robustness to variations in angles different from those used during training.

Angle

Point detection

Multimodal detection

shift

Global

Local

Global

Local

0

(82.0, 94.2)

(95.5, 84.4)

(86.9, 96.1)

(97.2, 90.2)

1

15

π

(82.6, 94.6)

(95.4, 83.9)

(86.7, 96.0)

(97.1, 90.7)

2

15

π

(82.1, 94.4)

(95.4, 84.1)

(86.4, 95.9)

(97.1, 90.7)

3

15

π

(82.6, 94.6)

(95.4, 84.3)

(86.4, 95.9)

(97.1, 90.7)

Lighting

Point detection

Multimodal detection

Global

Local

Global

Local

++

(82.0, 94.3)

(95.4, 83.7)

(85.7, 95.8)

(97.2, 90.7)

+

(82.4, 94.6)

(95.4, 83.8)

(86.1, 95.9)

(97.1, 90.5)

original

(82.0, 94.2)

(95.5, 84.4)

(86.9, 96.1)

(97.2, 90.2)

-

(82.4, 94.5)

(95.3, 83.9)

(86.4, 95.9)

(97.1, 90.6)

-

(81.9, 94.3)

(95.3, 83.4)

(86.1, 95.8)

(97.1, 90.5)

Lighting--

Lighting-

Original

Lighting+

Lighting++

Ground Truth

Original

Occlusion

Ground Truth

Figure 9: Visualization of occluded point clouds.

Rendering lighting Further exploring the robustness of PointAD under different conditions could enhance its generalized detection performance. We conducted ablation studies to test its sensitivity under different lighting conditions and with occluded point clouds below. To evaluate the impact of rendering lighting, we adjusted the lighting conditions to render point clouds, generating variant datasets with different lighting. We used both stronger and weaker lighting to render point clouds compared to the original dataset, covering a broad lighting range. We denote stronger and the strongest lighting as "+" and "++", and weaker and the weakest lighting as "-" and "-". Visualizations of the resulting samples are presented in Figure 8. The experiments were conducted on MVTec3D-AD, where we tested PointAD, trained on the original dataset, on these lighting variant datasets. Table 11 shows that PointAD can still detect anomalies even with significant discrepancies in rendering lighting, suggesting that PointAD is not sensitive to variations in rendering lighting.

Point occlusions Next, we evaluated detection performance with occluded point clouds. We occluded point clouds by removing those invisible from a specific rendering angle and then used the same rendering parameters to project the remaining points. During this process, we observed that abnormal regions might be occluded totally. Unlike class classification, where class semantics remain unchanged by point occlusions, removing these anomaly semantics would transform anomaly samples into normal points. Therefore, we selected rendering angles that allow visibility of part or all anomalous regions when the point cloud is an abnormal instance. The occluded point clouds are shown in Figure 9. We then used PointAD, trained on the original dataset, to test the resulting occluded dataset. Table 12 shows that PointAD suffers from performance degradation when the point clouds are occluded. We attribute this to two aspects: 1) Despite this strategy, the occluded

point clouds could still lose part of the anomaly semantics; 2) Occluded point clouds could create unexpected sinkholes on the surface, which may cause PointAD to identify these areas as hole anomalies incorrectly.

F Hyperparameter Ablation

Length of learnable prompts We study the sensitivity of important hyperparameters in PointAD. First, we explore the length of learnable text prompt templates, as shown in Table. 13. As the length of word embeddings increases, PointAD can better learn 3D and 2D anomaly semantics to improve its performance. Nonetheless, with a further increase in length (i.e., from 12 to 16), a decline in performance becomes noticeable. The excessive or insufficient number of learnable word embeddings can lead to performance degradation. An appropriate length ( i.e. , 12) is important for PointAD to attain comprehensive performance in both 3D and M3D anomaly detection.

Training set We have increased the test data for each category to incorporate more instances on MVTec3D-AD. Originally, we evaluated the rest of the data using only one category as the test data, such as carrot, cookie, and dowel. Now, we attempt to incorporate more instances. Here, we utilized two categories as auxiliary data, including carrot and cookies, carrot and dowel, and cookie and dowel. To further analyze the effect of the size of auxiliary data, we utilized three categories as auxiliary data, selecting all possible combinations from the four sets: carrot, cookie, bagel, and dowel. We present the performance averaged across all groups below. From Table 14 and Table 15, PointAD can incorporate more knowledge about abnormality and normality, improving point detection and multimodal detection performance. Specifically, from one category to three categories, PointAD exhibits improved performance, with P-AUROC increasing from 95.5%, 96.1%, to 96.3%, and AUPRO increasing from 84.4%, 86.3%, to 86.5%. Moreover, I-AUROC increases from 97.2%, 97.5%, to 97.8%, and AP increases from 90.2%, 91.8%, to 92.0%. This trend is also observed in global anomaly semantics.

G Failure Cases

In this section, we present the failure cases of PointAD, which we attribute to direct multimodality fusion. Since our model uses hybrid loss to incorporate the 3D and 2D anomaly semantics, it performs ZS multimodality 3D anomaly detection in a plug-and-play manner. However, when one modality prediction deviates severely from the ground truth in rare instances, direct fusion may result in an unpromising multimodal score map. As shown in Figure 10, the hole in the cookie is visually similar to the chocolate on cookies, making it challenging to differentiate the hole anomaly via color information alone. Although PointAD can detect the hole based on its abnormal point relations, the RGB score map heavily influences the final multimodal score map. Conversely, the tire presents an inverse situation where RGB can effectively predict the anomalies, but the point score map fails to recognize it. The false detection could arise from unusual point density and distribution. To demonstrate the effect of point density, we randomly select normal regions of point clouds and subsequently increase or decrease the density of these regions through upsampling and downsampling. We provide a qualitative analysis in 11. The visualization shows that PointAD effectively resists noise at reasonable levels. However, when noise levels are extremely low or high, the corresponding regions become excessively sparse or dense. This causes normal regions to appear similar to hole anomalies or squeezed anomalies, leading PointAD to classify these noisy areas as anomalies. Non-parametric score alignment and filter methods could be a potential direction, which we leave for further work.

10%

30%

100% (original)

1000%

3000%

Ground Truth

10%

30%

100% (original)

1000%

3000%

Ground Truth

Robustness to point density To investigate the impact of point density differences between the training and test datasets, we train PointAD using high-density point clouds (original dataset) and then test it on the low-density versions of the datasets, downsampled as described in Input resolution . Table 16 shows that PointAD can still detect anomalies even when retaining 50% of the points from the original point clouds. However, when more points are removed (30% and 20% sample ratio), PointAD experiences an obvious performance degradation. We attribute the misdetection to the overly sparse point clouds forming holes. Nevertheless, PointAD can still detect anomalies even with

Ground

truth

RGB

score map

Point

score map

Multimodal

score map

a significant gap in point density between the training and test domains (e.g., 100% vs. 20%). This demonstrates PointAD's ability to generalize across different point densities.

H Complexity analysis

In Table 17, we provide a comparison of computation overhead among unsupervised and zero-shot manners 4 . The evaluation includes inference time per image, frames per second (FPS), and GPU memory consumption with a batch size of 1. For a fair comparison, we keep NVIDIA RTX 3090 24GB GPU free until we conduct experiments. Compared to PointCLIP V2, our model requires less time to infer an image, achieving higher FPS (2.52 vs. 0.66) with lower graphic memory usage (as discussed in Section 3.5). While CLIP + rendering has a slight advantage in computation overhead, our detection performance significantly outperforms it. Therefore, PointAD achieves a favorable trade-off between performance and computation overhead.

Ground

truth

Point

score map

Multimodal

score map

Ground

truth

Point

score map

Multimodal

score map

Ours

PointCLIPV2

I Visualization Comparison

To provide intuitive results, we compare the visualization of PointAD with PointCLIP V2. In Figure 12, our model achieves accurate ZS 3D detection through the point cloud. Moreover, given RGB counterparts, PointAD further improves its detection capacity in M3D detection. However, PointCLIP V2 exhibits noisy activations for normal regions. After incorporating RGB information, PointCLIP V2 appears to struggle to fuse these two modalities in a plug-and-play manner, unlike PointAD.

J Additional Visualization

We respectively visualize the 2D renderings and corresponding 2D ground truths, which are rendered from 3D pint clouds and ground truths, as shown in Figure 13. We also supplement more zero-shot segmentation results of PointAD in Figure 14.

2D

renderings

2D ground

truth

2D

renderings

2D ground

truth

2D

renderings

2D ground

truth

2D

renderings

2D ground

truth

View 1

View 2

View 3

View 4

View 5

View 6

View 7

View 8

View 9

了

K Detailed results

We provide the class-level ZS 3D results on MVTec3D-AD in Table 18.

We provide the class-level ZS M3D results on MVTec3D-AD in Table 19.

We provide the class-level ZS 3D results on Eyecandies in Table 20.

We provide the class-level ZS M3D results on Eyecandies in Table 21.

We provide the class-level ZS 3D results on Real3D-AD in Table 22.

We provide the class-level ZS cross-dataset 3D results on Eyecandies from MVTec3D-AD in Table 23.

We provide the class-level ZS cross-dataset 3D results on Eyecandies from MVTec3D-AD in Table 24.

We provide the class-level ZS cross-dataset 3D results on Real3D-AD from MVTec3D-AD in Table 25.

Point

score map

Multimodal

score map

Ground

truth

Point

score map

Multimodal

score map

Ground

truth

Point

score map

Multimodal

score map

Ground

truth

Point

score map

Multimodal

score map

Ground

truth

.3

2

1.97.

7

Method

Candy

Cane

Chocolate

Cookie

Chocolate

Praline

Confetto

Gummy

Bear

Truffle

Licorice

Sandwich

Lollipop

Marsh-

mallow

Peppermint

Candy

Mean

MG.

PoinCLIPV2

a

(42.1, 50.4)

(45.5, 54.9)

(49.0, 52.5)

(57.1, 54.3)

(44.9, 45.8)

(53.6, 56.6)

(59.0, 64.5)

(47.4, 36.1)

(54.1, 53.3)

(32.6, 40.3)

(48.5, 50.9)

AnomalyCLIP

(44.9, 51.4)

(66.1, 68.9)

(76.4, 77.7)

(79.3, 82.7)

(51.0, 56.7)

(55.1, 58.3)

(75.2, 79.6)

(65.4, 57.9)

(70.3, 72.5)

(72.8, 75.5)

(65.7, 68.1)

PointAD-CoOp

(53.4, 58.6)

(72.8, 80.4)

(83.6, 86.6)

(89.1, 92.8)

(73.9, 76.2)

(72.0, 76.2)

(84.6, 88.4)

(65.7, 54.3)

(78.4, 83.3)

(89.6, 92.7)

(76.3, 78.9)

Point-AD

(49.4, 55.1)

(87.4, 91.2)

(87.5, 88.8)

(91.0, 94.1)

(71.4, 74.2)

(70.2, 75.0)

(88.0, 89.5)

(73.2, 63.5)

(79.4, 84.4)

(88.2, 91.9)

(78.6, 80.8)

ML.

PoinCLIPV2

a

(45.1, -)

(47.2, -)

(48.1, -)

(63.1, -)

(49.7, -)

(54.0, 15.6)

(45.2, -)

(33.8, -)

(44.7, -)

(41.9, -)

(47.3, -)

AnomalyCLIP

(95.1, 82.3)

(91.6, 71.2)

(84.3, 62.0)

(94.1, 80.9)

(80.6, 44.5)

(73.6, 42.7)

(89.6, 66.9)

(92.6, 68.0)

(75.9, 42.8)

(84.9, 52.1)

(86.2, 61.3)

PointAD-CoOp

(92.0, 71.0)

(96.8, 86.4)

(94.2, 86.2)

(97.9, 92.5)

(87.9, 64.8)

(91.3, 68.6)

(96.2, 83.6)

(96.5, 82.5)

(95.3, 81.3)

(96.1, 86.7)

(94.4, 80.3)

Point-AD

(88.8, 63.9)

(97.4, 88.9)

(92.5, 85.1)

(99.1, 96.6)

(89.9, 70.4)

(89.9, 70.4)

(95.5, 84.4)

(95.7, 79.8)

(95.8, 82.5)

(95.2, 84.8)

(94.0, 80.7)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction precisely reflect the contribution and scope of this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: we have created a separate "Limitations" section in our paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a detailed illustration of our proposed algorithm and baselines in the Appendix C, Appendix B, and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: we will make our code and dataset available once the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide full details in Appendix B and Appendix F

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the average results across three runs in Section 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We point out the specific compute resources in Section 4.2

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper obeys the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts of our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: our paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: we respect the Licenses for existing assets that we use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: we will release new assets proposed in our paper once the paper is accepted. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection\" introduces PointAD, a novel framework designed for zero-shot 3D anomaly detection by leveraging both point clouds and RGB images. The approach builds on the strong generalization capabilities of CLIP and adapts it to 3D anomaly detection. The framework renders 3D anomalies into multiple 2D views and then projects these 2D representations back into 3D space. This hybrid representation learning allows for capturing generic anomaly semantics across various unseen 3D objects. PointAD also incorporates auxiliary point clouds to optimize learnable text prompts for better anomaly detection and segmentation.\n\nSTRENGTHS:\n1. The use of CLIP for 3D anomaly detection and the hybrid representation learning approach are novel contributions.\n2. The theoretical foundations are robust, and the empirical validation is comprehensive.\n3. The paper is clearly written, with well-structured sections and effective use of visual aids.\n4. The approach addresses a critical gap in the field, with potential for significant impact on future research and practical applications.\n\nWEAKNESSES:\n1. The sensitivity of the method to the selection of rendering angles and the number of views could be explored further.\n2. While the experimental results are compelling, additional validation on more diverse and challenging datasets would further strengthen the claims.\n3. The discussion of potential limitations and future work could be expanded to provide a more comprehensive view of the method's applicability and areas for improvement.\n\nQUESTIONS:\n1. Could the authors provide more insights into the selection of rendering angles and the impact of different numbers of views on the detection performance?\n2. Have the authors considered the robustness of the method under different environmental conditions, such as varying lighting and occlusions?\n3. Could additional experiments on other industrial or medical datasets help to further validate the generalizability of the proposed method?\n\n\n======== post rebuttal ==========\nThe authors' rebuttal solve most of my concerns, hence I raise my score to borderline accept.\n\nLIMITATIONS:\nThe authors have addressed some limitations, including the fixed rendering angles and the potential for further optimization of view selection. Constructive suggestions for improvement include exploring the sensitivity to rendering parameters and validating the method under different environmental conditions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PointAD, the first approach to explore the domain of zero-shot 3D anomaly detection, leveraging CLIP's strong generalization to identify anomalies in previously unseen objects. It offers a unified framework that integrates 3D point clouds with 2D renderings, employing hybrid representation learning to capture the semantics of anomalies. PointAD's key contributions are its pioneering exploration of the ZS 3D anomaly detection domain, its ability to seamlessly integrate multimodal data like RGB information for enhanced detection, and its superior performance over existing methods. The robustness of the framework is confirmed through extensive experiments.\n\nSTRENGTHS:\nOriginality: The paper introduces PointAD, a novel method in 3D anomaly detection that leverages CLIP for 3D analysis, uniquely combining point clouds and pixel data. It expands the application of vision-language models into new domains, showcasing versatility in 3D point cloud analysis.\nSoundness: The paper exhibits methodological soundness through its rigorous experimental setup, including the use of diverse datasets and a thorough ablation study that substantiates the design decisions and effectiveness of the proposed PointAD framework. The state-of-the-art experiments performance confirms the model's outperforming ability.  \nClarity: The paper stands out for its clarity, guiding readers smoothly from the problem statement to the final results. It skillfully explains complex concepts in an accessible way, ensuring that a wider audience can follow along. The paper also benefits from helpful visual aids that clearly demonstrate the model's effectiveness.\n\nWEAKNESSES:\n1.\tThere’s some zero-shot anomaly detection methods with clip on 2D images, so this paper should compare with them, especially, from the view of techniques’ differences, not only the used data differences. For example, those zero-shot methods (WinCLIP, VAND and AnomalyCLIP) mentioned in [R1] and [R2].\n[R1] Cao, Y., Xu, X., Zhang, J., Cheng, Y., Huang, X., Pang, G., & Shen, W. (2024). A survey on visual anomaly detection: Challenge, approach, and prospect. arxiv preprint arxiv:2401.16402.\n[R2] Li, X., Huang, Z., Xue, F., & Zhou, Y. (2024). Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images. arxiv preprint arxiv:2401.16753.\n\n2.\tI also wonder if the new methods of 3D anomaly detection can also be used to deal with this zero-shot task. For example, [R3] and [R4]. \n[R3] Zuo, Z., Dong, J., Wu, Y., Qu, Y., & Wu, Z. (2024). CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation. arxiv preprint arxiv:2406.18941.\n[R4] Li, W., Xu, X., Gu, Y., Zheng, B., Gao, S., & Wu, Y. (2024). Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22207-22216).\n3.\tMore importantly, there’s some very similar studies with this work and task. Authors should mention and compare them within this study. For example, [R5].\n[R5] Li, Y., Goodge, A., Liu, F., & Foo, C. S. (2024). Promptad: Zero-shot anomaly detection using text prompts. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1093-1102).\n[R6] X. Chen, J. Zhang, G. Tian, H. He, W. Zhang, Y. Wang, C. Wang,Y. Wu, and Y. Liu, “Clip-ad: A language-guided staged dual-\npath model for zero-shot anomaly detection,” arXiv preprint\narXiv:2311.00453, 2023.\n[R7] Wang, C., Zhu, H., Peng, J., Wang, Y., Yi, R., Wu, Y., ... & Zhang, J. (2024). M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising. arxiv preprint arxiv:2406.02263.\n4.\tComputational Efficiency: Although the authors have commendably addressed computational challenges, the overall computational cost associated with rendering multi-view images and processing them through the vision encoder is still substantial. Since this field is still developing with few comparable studies, it is recommended that the authors provide a broader comparison about the computational cost with existing anomaly detection techniques, including those like Cheraghian, CPFM and 3DSR that are not strictly within the zero-shot category. This expanded comparison will help to show that the computational demands of PointAD are reasonable and suitable for practical use cases.\n5.\tClarity and Completeness of Presentation: The paper's writing is generally clear and free of errors, yet minor aspects of the presentation could be polished for better reader comprehension. For instance, the figure overlays, specifically in Figure 1, currently only elucidate part (a), leaving explanations for parts (b) and (c) to be found in the main text. This separation can disrupt the reader's flow and understanding. Enhancing the figure overlays to include all relevant parts and ensuring that each component of the figure is self-explanatory will make the paper more accessible and its findings easier to grasp.\n\nQUESTIONS:\n1.\tPlease answer the main differences with these mentioned methods in Weaknesses and compare with part of them on experiments.\n2.\tGiven the concerns raised in the Weaknesses section 1, does PointAD still hold the merit of computational efficiency when compared to other methods such as AnomalyCLIP and Cheraghian?\n3.\tCould you elaborate on the multi-view rendering process of point clouds (as described from line 134 to line 140), particularly how the color of the rendered images is determined? Furthermore, at line 254, it's stated that additional RGB information is utilized only in the test dataset. Given the description at line 564, which refers to 2D RGB information, how is this 2D data integrated with the rendered multi-view images? Are there any strategies which are employed to bridge the gap between the rendered training data and the RGB information used in testing?\n\nLIMITATIONS:\nThe authors have discussed the limitations of their model, and there are no potential negative societal impacts associated with their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose a unified framework, PointAD,to detect 3D anomalies in a ZS manner. Hybrid representation learning is proposed to incorporate the generic normality and abnormality semantics into PointAD. PointAD can incorporate 2D RGB information in a plug-and-play manner for testing, which can perform ZS M3D anomaly detection directly. Experiments results demonstrate the superiority of the model in detecting and segmenting 3D anomalies.\n\nSTRENGTHS:\n1. PointAD introduces a novel method for Zero-Shot 3D anomaly detection by integrating the strong generalization capabilities of CLIP to recognize 3D anomalies from unseen objects. This approach is first of its kind in ZS 3D anomaly detection.\n2. The paper describes a unified framework that comprehends 3D anomalies from both points and pixels, allowing for a comprehensive understanding of anomalies by rendering 3D objects into 2D renderings and projecting them back to 3D space. It proposes hybrid representation learning to optimize learnable text prompts from both 3D and 2D data, enhancing the model's ability to generalize across diverse unseen objects.\n3. Extensive experiments demonstrate the superiority of PointAD over existing state-of-the-art methods, especially in ZS settings where traditional models struggle.\n\nWEAKNESSES:\nI do not notice any obvious weaknesses in this paper, but I still have a few questions that I hope the authors can answer.\n1. The methodology, while innovative, is complex due to the need to render 3D objects into multiple 2D views and then re-project them, the performance of the model may heavily depends on the quality of the 2D renderings of the 3D objects, Therefore, if the renderings are of poor quality, could the model's performance degrade?\n2. The experiments in the paper utilize high-resolution point cloud scans. Given that the point clouds obtained in practical applications may not always be high resolution, how effective is this method in detecting anomalies in low-resolution, sparse scans?\n3. In the appendix's failure case section, the detection of the tire failed on the point score map. Thus, if a tire, or any object, has noise, how significantly does it affect the recognition on the point score map? I suspect that the possible reason for this failure could be the algorithm's sensitivity to point density and spatial distribution. When anomalies are located in areas where point density or distribution are inconsistent or significantly different from the training data, can this model still generalize well to new, unseen anomalies?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors adequately addressd the limitations. Societal impact is not discussed, but I believe it's not needed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}