PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection

Authors: qihang  zhou
jiangtao  yan
shibo  he
wenchao  meng
jiming  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:14:59

Abstract: Zero-shot (ZS) 3D anomaly detection is a crucial yet unexplored field that addresses scenarios where target 3D training samples are unavailable due to practical concerns like privacy protection. This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD provides a unified framework to comprehend 3D anomalies from both points and pixels. In this framework, PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. To capture the generic anomaly semantics into PointAD, we propose hybrid representation learning that optimizes the learnable text prompts from 3D and 2D through auxiliary point clouds. The collaboration optimization between point and pixel representations jointly facilitates our model to grasp underlying 3D anomaly patterns, contributing to detecting and segmenting anomalies of unseen diverse 3D objects. Through the alignment of 3D and 2D space, our model can directly integrate RGB information, further enhancing the understanding of 3D anomalies in a plug-and-play manner. Extensive experiments show the superiority of PointAD in ZS 3D anomaly detection across diverse unseen objects.

Main paper: 
 introduction: 
 Anomaly detection, a significant field within deep learning, has been widely applied to diverse domains, including industrial inspection [2, 3, 32, 36, 37, 44, 15, 24, 55, 6, 19, 63] . While 2D anomaly detection has been extensively studied by exploring RGB information [23, 56, 50, 51, 8, 29] , real-world anomalies typically present themselves with abnormal spatial characteristics. Relying solely on RGB information poses challenges in detecting some anomalies in many cases, e.g., when the defect mimics the appearance of the object's background or foreground, as shown in Figure 1(a) . The emerging field of 3D anomaly detection aims to unveil these spatial relations indicative of abnormal patterns [22, 4, 45, 9, 54, 13] .However, current 3D anomaly detection methods typically store normal point features during training and identify anomalies by measuring the distance between the test feature and these stored features [22, 54, 13] . They all depend on the assumption that target point clouds are available and entirely normal. This assumption does not hold in various situations when the training samples in the target dataset are inaccessible due to privacy protection (e.g., involvement of trade secrets) or the absence of target training data (e.g., a new product never seen before) [63] . Figure 1 (b) depicts the setting discrepancy between ZS 3D and unsupervised anomaly detection. These methods mentioned above, which detect anomalies by memorizing or reconstructing normal point features, have limitations in generalizing to unseen objects in Figure 1(c ). While zero-shot (ZS) anomaly detection has been explored in 2D images [35, 63] , ZS 3D anomaly detection remains a research blank. It is a challenging task as ZS Figure 1 : Motivation of zero-shot 3D anomaly detection. (a): Top: The hole on the cookies presents a similar appearance to the background. Bottom: Surface damage on the potato is unapparent to the object foreground. In these cases, leveraging RGB information makes it difficult to detect anomalies that imitate the color patterns of the background or foreground. However, effective recognition can be achieved by modeling the point relations within corresponding point clouds. (b) and (c) depicts the setting difference of ZS and unsupervised manner.3D anomaly detection necessitates the model to detect 3D anomalies across unseen point clouds with diverse class semantics, requiring a robust generalization capacity in the detection model. Recently, Vision-Language Models (VLMs) with their strong generalization capabilities have been applied to various downstream tasks [40, 61, 41, 49, 25, 26] . Particularly, CLIP has demonstrated its strong ZS performance to detect 2D anomalies [35, 24, 63] . Integrating CLIP into the detection model presents a potential solution to the challenging yet unexplored ZS 3D anomaly detection.In this paper, we propose a unified framework, namely PointAD, to transfer the knowledge of CLIP to detect 3D anomalies in a ZS manner. PointAD comprehends point clouds from both 3D and 2D:(1) deriving 2D representations of point clouds via CLIP by rendering them from multiple views, (2) understanding 3D representations by projecting 2D representations back to 3D, and (3) enhancing 3D comprehension by additional regularization on 2D representations. After grasping point clouds from points and pixels, we propose hybrid representation learning to capture generic normality and abnormality w.r.t. point and pixel information into learnable text prompts [63] . Specifically, since 3D representation manifests its 2D renderings from different views, we treat each representation as one instance and achieve 3D representation aggregation via multiple instance learning (MIL). On this basis, PointAD explicitly aligns the 2D anomalies, rendered from 3D anomalies, to further enhance 3D understanding. We formulate these 2D anomaly recognition tasks from the multi-task learning (MTL) perspective. PointAD collaboratively learns point and pixel representations, promoting the in-depth understanding of underlying abnormal patterns and thus achieving superior ZS normality and abnormality point recognition. Furthermore, benefiting from collaboration optimization, PointAD can directly integrate additional RGB information and perform ZS multimodal 3D (M3D) detection without extra modules and retraining. The main contributions of this paper are summarized as follows:• To the best of our knowledge, we are the first to investigate the challenging yet valuable ZS 3D anomaly detection domain. We propose to transfer the strong recognition generalization of CLIP to detect and segment 3D anomalies over diverse objects. • We introduce a novel ZS 3D anomaly detection approach called PointAD, which provides a unified framework to understand 3D anomalies from points and pixels. Hybrid representation learning is proposed to incorporate the generic normality and abnormality semantics into PointAD, enabling a thorough understanding of 3D anomalies. • PointAD can incorporate 2D RGB information in a plug-and-play manner for testing. In contrast to other methods that require storing RGB information separately, PointAD offers a unified framework to perform ZS M3D anomaly detection directly. • Extensive experiments are conducted to demonstrate the superiority of our model in detecting and segmenting 3D anomalies, even outperforming some unsupervised SOTA methods that memorize normal features of target objects in certain metrics. We hope that our model will serve as a springboard for future research on ZS 3D and M3D anomaly detection. 
related work: 
 3D Anomaly Detection MVTec3D-AD [4] , Eyecandies [5] , and Real3D-AD [31] provide the point cloud anomalies and the corresponding 2D-RGB information. MVTec3D-AD bridges the connection between 3D and 2D anomaly detection. 3D-ST [4] uses a teacher net to extract dense local geometric descriptors and design a student net to match such descriptors. AST [45] introduces asymmetric teacher and student net to further improve 3D anomaly detection. IMRNet [28] and 3DSR [58] detect 3D anomalies by reconstruction errors. Instead of only using point clouds, BTF [22] , M3DM [54, 53] , CPFM [7] , and SDM [13] integrate point features and RGB pixel features to detect 3D anomalies. While these approaches exhibit commendable performance by storing object-specific normal point and pixel features within the unsupervised learning framework, such paradigms simultaneously limit their generalization capacity to point clouds from unseen objects, which is crucial to detecting anomalies when the target object is unavailable. To the best of our knowledge, no solution addresses this valuable yet challenging problem. To fill this gap, we introduce PointAD, designed to identify unseen anomalies across diverse objects. PointAD extends CLIP to the realm of ZS 3D anomaly detection and shows robust generalization in capturing generic normality and abnormality within point clouds. Furthermore, PointAD serves as a unified framework, allowing seamless integration of point cloud and RGB modality without additional training.3D Feature Extraction Conventional methods of 3D feature extraction typically employ a pointbased network like PointNet [38] or PointNet++ [39] to extract 3D features from point clouds.Alternative approaches convert 3D data into a 2D format [48, 18] , enabling 2D image backbones to process 3D information. PointCLIP [59] directly projects raw points onto image planes for efficiency, but this approach causes the produced depth map to lack geometric details. Instead, rendering-based methods [48, 21] generate 2D renderings by rendering point clouds, allowing for better preservation of local semantics. CPFM [7] stores normal features of these 2D renderings for unsupervised 3D anomaly detection. In this paper, we apply this rendering strategy to the source samples to capture generic anomaly semantics for recognizing abnormalities in unseen objects.Prompt Learning Instead of fine-tuning the whole network, prompt learning just optimizes the model to adapt the network to downstream tasks. CoOp [61, 60] introduces global context optimization to update learnable text prompts for few-shot recognition. DenseCLIP [41] extends it to the dense classifications. More recently, AnomalyCLIP [63] proposes object-agnostic prompt learning to capture the generic normality and abnormality for images. Our model first introduces hybrid representation learning for ZS 3D anomaly detection, enabling the detection of anomalies and abnormal regions. EQUATIONwhere cos(•, •) and τ represent the cosine similarity and temperature used in CLIP, respectively. The segmentation S i(c) ∈ R h×w for class c can be computed as Seg(g c , f m i ), where each entry (u,v) is calculated as P (g c , f m i,u,v ). 
overview of pointad: 
 ZS 3D anomaly detection requires a strong generalization capacity to anomalies on unseen objects with diverse object semantics. In this paper, we propose a unified framework, namely PointAD, to detect and segment 3D anomalies in a ZS manner. In Figure 2 , PointAD understands point clouds from both pixel and point perspectives. To make CLIP understand 3D point clouds, we first render point clouds from multiple views and extract the pixel representations of these generated 2D renderings via the visual encoder of CLIP. And then, we derive point representations by projecting these pixel representations back to 3D. Learning generic normality and abnormality is significant in recognizing across-object anomalies. We propose hybrid representative learning, which focuses on glocal point and pixel abnormality, to optimize normality and abnormality text prompts, enabling PointAD with strong generalization to identify 3D anomalies on diverse objects. Benefiting from the hybrid representation learning, PointAD can directly incorporate 2D RGB information during testing to achieve ZS M3D detection. … W 1 W E damaged object or … V 1 V E class … V 1 V E object or 1 0.95 1 0.92Multi-view projection is a crucial technology for understanding point clouds from 2D perspectives. Some multi-view projection approaches project point clouds into various depth maps, providing adequate shape information for class recognition [59] . However, in this paper, our objective is to learn both generic global and local anomaly semantics. Depth-map projection lacks sufficient resolution to represent fine-grained anomaly semantics accurately. Hence, we adopt high-precision rendering to preserve the original 3D information offline. Specifically, given an auxiliary dataset of point clouds D 3d = {(x 3d i , y 3d i )} N i=1 , we define the rendering matrix as R (k) for the k-th view, with a total of K views. We simultaneously render point clouds and point-level ground truths from different views to obtain their corresponding 2D renderings, which is given by x(k) i = R (k) (x 3d i ) and y (k) i = R (k) (y 3d i ), where x (k) i ∈ R H×W and y (k) i∈ R H×W respectively represent the k-th 2D renderings and corresponding pixel-level ground truth in the i-th point cloud. Note that anomaly pixels are marked as 1, and normal pixels are marked as 0.PointAD aims to learn generic anomaly semantics from both 3D and 2D representations, enabling a comprehensive understanding of point and pixel anomaly patterns. For a point cloud x 3d i , we first obtain the 2D renderingsX i = {x (k) i } K k=1 .Then, these renderings are encoded via the vision encoder of CLIP to obtain global 2D representationsF i ={f (k) i } K k=1 ,F m i = {f m(k) i } K k=1 .As for point cloud representations, we consider that one point cloud will be projected into multiple 2D renderings. Consequently, global 3D representation p i and local 3D representations p m i are expected to include their corresponding 2D representations in each view. Formally,p i = {p (k) i |p (k) i = f (k) i } K k=1 and p m i = {p m(k) i |p m(k) i = {p m(k) i,j } n j=1 } K k=1 , where p m(k) i,j = {p m(k) i,j = f m(k) i,u,v , (u, v) = R ′(k) (a i,j |b i,j , c i,j )} represents the j-th point representation of i-th point cloud in the k-th view, whose 3D coordinate is (a i,j , b i,j , c i,j ). R ′(k) is the rendering transformation between the point and pixel representation, derived as R ′(k) = h H R (k) .Points at different positions may yield a different number of 2D representations as they are hidden by other points from a specific viewpoint. In this case, we introduce a view-wise visibility mask M , where M k i,j indicates whether the j-th point of the i-th point cloud is visible in the k-th view. We compare the depth of points projected into the same pixel in the same view and set the corresponding visibility mask to 1 for the point with the minimum depth, and to 0 for the other points. Let Q(k) i,u,vdenote the depths set of all points that are projected into the same pixel indexed by (u, v) in the i-th point cloud in the k-th view. Q (k) i,u,v and M k i,j are respectively given asQ (k) i,u,v = {c i,j | R ′(k) (a i,j , b i,j , c i,j ) = (u, v)} n j=1 and M (k) i,j = I(i, j, k = argmin i,j,k {c i,j | c i,j ∈ Q (k) i,u,v }), where I(•) is an indicator function. Local 3D representations of the i-th point cloud for the k-th view are reformulated as:p m(k) i = {p m(k) i,j * M (k) i,j } n j=1 .The key of ZS 3D anomaly detection requires the model to capture generic anomaly semantics, rather than relying on specific object semantics. [61] , to obtain more overall text embeddings g n and g a to recognize normality and abnormality, respectively.tn = [V1] . . . [VE][object], tn = [V1] . . . [VE][class], ta = [W1] . . . [WE][damaged][object] PointAD , ta = [W1] . . . [WE][damaged][class] PointAD-CoOp, where V and W are learnable word embeddings, respectively.PointAD from point information, we respectively devised two losses to capture 3D global anomalies and local anomaly regionals. First, we compute the cosine similarity between the textual representation and its rendering global representations in each view. As point clouds are projected from different views, the resulting renderings in each view reflect certain parts of point clouds. We use view-wise MIL to integrate 2D global representations and then align global labels to capture the global semantics. Formally, the global 3D loss is defined as:EQUATIONi ), max (y 3d i )).As for local point anomaly semantics, we quantify the cosine similarity between textual representations and local representations of 2D renderings. Since points within point clouds are projected from different views, their projections in each view present part characteristics of themselves. We adopt the pixel-wise MIL to achieve the aggregation of point local representation. The point segmentation can be formulated mathematically as follows:S 3d i(a) = 1 K k Seg(ga, p m(k) i ), S 3d i(n) = 1 K k Seg(gn, p m(k) i).However, deriving such 3D segmentation requires similarity computation for each point. It brings a significant memory burden, with a huge computational complexity of O(Knd), which is unaffordable for one NVIDIA RTX 3090 24GB GPU. To address this computational challenge, we resort to the rendering correspondence between points (3D space) and their corresponding pixels within each view (2D space). We first can rewrite 3D segmentation from the view perspective asS 3d i(a) = 1 K k S 3d(k) i(a). Then, the k-th division of 3D segmentation can be transformed into the 2D counterpart through the rendering projection S 3d(k)i(a) = (R (k) ) (-1) S 2d(k) i(a) ⊗ M (k) i , where ⊗ is the Hamiltonian product. The k-th 2D counterpart can be computed as S 2d(k) i(a) = Up(Seg(g a , f m(k) i)), where the operator Up(•) represents bilinear interpolation from feature space to 2D space. Finally, we can reformulate the 3D segmentation as follows:EQUATIONFrom the equation, we can observe that the primary computation can be conducted in the feature space, with a computational complexity of O(Khwd). This is a substantial overhead reduction compared to O(Knd) since feature space is much smaller than 3D space, i.e., h × w ≪ n. In our experiment, h × w = 24 × 24 = 576, while n = 336 × 336 = 112896. With this transformation, the : is defined as:L local 3d = 1 N i Dice(S 3d i(n) , I -y 3d i ) + Dice(S 3d i(a) , y 3d i ) .EQUATIONi ), max (y(k) i )).Also, we focus on 2D abnormal regions to understand pixel-level anomalies. As the anomaly regions are typically smaller than normal regions, we employ Focal Loss to mitigate the class imbalance besides Dice Loss. Let ⊕ denote the concatenation operation. Local MTL-based 2D representation learningL local 2d is given as follows:EQUATIONi ).PointAD detects 3D anomalies from both 3D and 2D perspectives and thus combing these above losses to derive hybrid loss L hybrid . We minimize L hybrid to incorporate generic anomaly semantics into the text prompt from point and pixel spaces:L hybrid = L global 3d + L local 3d + L global 2d + L local 2d .During training, we minimize the hybrid loss L hybrid , where the original parameters of CLIP are frozen to maintain its strong generalization. Since our model provides a unified framework to understand anomaly semantics from point and pixel, it can not only perform ZS 3D anomaly detection but also M3D anomaly detection in a plug-and-pay way. Next, we will introduce the inference process in detail: ZS 3D/M3D inference Given a point cloud x 3d i , we regard the 3D segmentation (See Equ. 2) as the anomaly score map:A m i =G σ (S 3d i(a)), where G σ (•) represents the Gaussian filter. The global anomaly score incorporates glocal anomaly semantics and is computed asA s i = 1 2 ( 1 K f (k) i ∈Fi P (g c , f (k) i ) + max (A m i )).When the RGB counterpart is available for testing, PointAD could directly integrate RGB information by feeding RGB images to 2D branch to derive 2D representations. We project these 2D representations back to 3D branch to respectively compute RGB anomaly score map and anomaly score asA m(rgb) i = P (g c , f (rgb) i ) and A s(rgb) i = G σ (S 3d(rgb) i(a)). The final multimodal anomaly score map and anomaly score are defined asA m(mod) i = 1 2 G σ A m i + A m(rgb) i and A s(mod) i = 1 2 1 2 (A s(rgb) i + A s i ) + max (A m(mod) i) , respectively. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [No] Justification: we will make our code and dataset available once the paper is accepted. Guidelines: • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impacts of our paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to Answer: [NA] Justification: we will release new assets proposed in our paper once the paper is accepted. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://github.com/isl-org/Open3Dhttps://github.com/mlfoundations/open_clipSince the official code for WinCLIP is not available, we reproduce its results using the implementation found at https://github.com/zqhang/Accurate-WinCLIP-pytorch 
implementation details & baselines: 
 Both point clouds and 2D renderings are resized to 336 × 336. We use Open3d library to generate 9 views by rotating point clouds along the X-axis at angles of {- 4 5 π, -3 5 π, -2 5 π, -1 5 π, 0, 1 5 π, 2 5 π, 3 5 π, 4 5 π} for most categories. We circularly set the rendering angles, evenly distributing the angles between -π to π. The backbone of PointAD is the pre-trained CLIP model (VIT-L/14@336px in open_clip). Following [63], we improve the local visual semantics of vision encoder of CLIP without modifying its parameters. During training, we keep all parameters of CLIP frozen and set the learnable word embeddings in object-agnostic text templates to 12. All experiments were conducted on a single NVIDIA RTX 3090 24GB GPU using PyTorch-2.0.0. As there is no work to explore the field of ZS 3D anomaly detection, we make a great effort to provide these comparisons. We apply the original CLIP to our framework for 3D detection, called CLIP + Rendering. Also, we reproduce SOTA 3D recognition works including PointCLIP V2 [64] and Cheraghian [12] , and adapt them for ZS 3D anomaly detection. We compare the SOTA 2D anomaly detection approach AnomalyCLIP [63] by fine-tuning it on depth maps. PointAD by default uses object-agnostic text prompts, whereas PointAD-CoOp employs object-aware prompts. Appendix B and C provide more details on implementation and baselines.We fine-tuned PointAD on three objects on MVTec3D-AD, Eyecandies, and Real3D-AD. Over three runs, the averaged results on one-vs-rest and cross-dataset settings are reported. We use the metric pairs (I-AUROC% % ↑ and AP% % ↑) and (P-AUROC% % ↑ and AUPRO% % ↑) to evaluate the glocal detection performance, respectively. Details of experimental settings see Appendix A We also compare the ZS M3D anomaly detection when RGB information is available for testing. As shown in Table 2 , the results indicate that PointAD can integrate additional RGB information and further boost its performance from 82.0% to 86.9% AUROC and from 94.2% to 96.1% AP for global semantics on MVTec3D-AD. Additionally, as for local semantics, the performance improves from 95.5% to 97.2% P-AUROC and from 84.4% to 90.2% AUPRO. A large performance gain is also obtained on Eyecandies and Real3D-AD. While other methods improve their performance in some metrics, they still suffer from performance degradation in other metrics due to inefficient integration of the two modalities. Instead, PointAD achieves overall improvement across all metrics by incorporating explicit joint constraints on both point and pixel information. We compare all baselines that need fine-tuning. From Table 3 and M3D from Table 4 , PointAD demonstrates strong cross-dataset generalization performance on Eyecandies and Real3D-AD, with nearly no obvious performance decay compared to the one-vs-rest setting. The strong transfer ability highlights its robust generalization capabilities in detecting anomalies in objects with unseen semantics and backgrounds.Visualization analysis. To intuitively present the strong generalization capacity of our model to unseen anomalies, we visualize the anomaly score maps of the 3D and corresponding 2D counterparts of PointAD on MVTec3D-AD. As shown in Figure 3 , PointAD reveals abnormal spatial relationships of points and further captures the generic point anomaly patterns across diverse objects. And, we also visualize the anomaly score map of corresponding 2D counterparts, where 3D point anomalies are transformed into 2D pixel anomalies. It can be observed that PointAD also has a strong detection ability on such 2D anomalies. The strong representative pixel representations from multiple views facilitate more precise 3D anomaly detection. Quantitative results are provided in Section 5. The strong 3D and 2D detection capabilities of PointAD are from hybrid representation learning, which not only enables PointAD to capture the 3D anomalies but also explicitly constrains 2D representations. How multimodality makes PointAD accurate. PointAD is a unified framework that can not only capture point anomalies but also handle 2D information in a plug-and-play manner. As shown in Figure 4 (a), we visualize M3D results of PointAD on MVTec3D-AD. The surface damage on the potato presents a similar appearance to the object foreground, which makes it difficult to detect this anomaly with RGB information. On the contrary, the point relations for the color stain on foam are the same as those of normal, but they have a clear distinction in the RGB information. PointAD can integrate these two modalities, thereby complementing their respective advantages. We further investigate the reason why PointAD can directly leverage both modalities. For this purpose, we experiment without 2D glocal loss. As shown in Figure 4 (b), without 2D glocal loss, significant noise disrupts and even covers the RGB score maps, resulting in unpromising multimodal fusions. This illustrates the importance of explicit constraints on the 2D space. Hence, we conclude that the robust multimodal detection capability of our model stems from the collaboration optimization in both 3D and 2D spaces during training. We provide more analysis about failure cases and the computation overhead in Appendix G and H. 
ablation study: 
 Module Ablation. Here, we investigate the effectiveness of the proposed main technologies by progressively adding the proposed modules. Broader Impact Our paper aims to enhance automated detection and decision-making in smart manufacturing, which does not involve any potential ethical risks. Since the collection of 3D samples is more labor-intensive and costly, our research on using vision-language models for zero-shot point cloud detection can have significant societal impacts, especially in scenarios where target 3D training samples are unavailable due to privacy concerns or the absence of products. We hope that our first exploration of the ZS 3D anomaly detection field could pave the way for further research in this emerging field.[63] Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Objectagnostic prompt learning for zero-shot anomaly detection, 2023.[ Point-wise annotations are available for the point clouds. MVTec3D-AD also provides corresponding 2D-RGB image counterparts for the point clouds. We remove the background plane of point clouds in the whole dataset like [22] . Eyecandies also has 10 different classes and provides the corresponding 2D RGB information. Real3D-AD is a recently available dataset, which contains 12 objects. However, it does not provide the RGB information.Evaluation Setting and Metric Since the training dataset of MVTec3D-AD only contains all normal samples, we use an object test dataset as the auxiliary dataset to fine-tune PointAD and assess the ZS anomaly detection for the remaining objects. In particular, we report the average results using different objects as the auxiliary, i.e., carrot, cookie, and dowel for MVTec3D-AD; confetto, LicoriceSandwich, and PeppermintCandy for Eyecandie; seahorse, shell, and starfish for Real3D-AD. Moreover, we explore more challenging cross-dataset generalization settings, where we use auxiliary data to test all objects of another dataset. For point cloud anomaly detection, we only use point clouds to detect and localize 3D anomalies in Figure 5 (a). In M3D anomaly detection, both point clouds and their 2D-RGB counterparts are utilized, as shown in Figure 5 (b). To comprehensively analyze PointAD, we utilize four metrics to assess its anomaly classification and segmentation performance. For anomaly detection, we use the Area Under the Receiver Operating Characteristic Curve (I-AUROC% ↑) and average precision (AP% ↑). Regarding anomaly segmentation, we use point-level AUROC (P-AUROC% ↑) and a restricted metric called AUPRO%(↑) [3] to provide a detailed evaluation of subtle anomaly regions. Table 7 : Ablation study on the number of rendering views.Rendering angles1 0 3 -1 2 π, 0, 1 2 π 5 -2 3 π, -1 3 π, 0, 1 3 π, 2 3 π 7 -3 4 π, -1 2 π, -1{-4 5 π, -3 5 π, -2 5 π, -1 5 π, 0, 1 5 π, 2 5 π, 3 5 π, 4 5 π} for most categories. Some categories lose their surface completely because they are not stereo point clouds. Table 7 also gives the specific rendering angles of different view number settings. We set the rendering angles in a circular manner, evenly distributing the angles between -π to π. The backbone of PointAD is the pre-trained CLIP model 3 (VIT-L/14@336px). Following [63], we improve the local visual semantics of the vision encoder of CLIP without modifying its parameters and introduce learnable tokens in the text encoder. During training, we keep all CLIP parameters frozen and set the learnable word embeddings in objectagnostic text templates to 12. We use the Adam optimizer with a learning rate of 0.001 to optimize the learnable parameters. The experiment runs for 15 epochs with a batch size of 4. All experiments were conducted on a single NVIDIA RTX 3090 24GB GPU using PyTorch-2.0.0.ZS 3D anomaly detection and M3D anomaly detection have not yet been explored. We first make an adaption for the original CLIP for 3D anomaly detection. Then, we reproduce SOTA ZS 3D classification methods (i.e., PointCLIP V2 [64] and Cheraghian [12] ) and adapt them to our settings. SOTA unsupervised 3D anomaly detection approaches are reported as the performance upper bound. All hyperparameters in these baselines are kept the same. We will present the detailed reproduction as follows:• CLIP + Rendering is a method, where we apply the original CLIP into our framework for ZS 3D anomaly detection. It uses the same rendering procedure as PointAD. Following [24, 63] • AnomalyCLIP (ICLR 2024) is a SOTA zero-shot 2D anomaly detection method. Anomaly-CLIP introduces object-agnostic learning to capture generic anomaly semantics of images. We adapt AnomalyCLIP in 3D detection by fine-tuning AnomalyCLIP on depth images of corresponding point clouds.• Cheraghian (IJCV 2022) is an approach for ZS 3D anomaly detection without foundation models. They directly extract the point presentations by PointNet and use word2vector [34] to generate the textual embedding of an object. To incorporate the anomaly semantics into Cheraghian, we respectively average the textual embedding of an object name and damaged.We replace the global representation with dense representations to provide the segmentation results.2D anomaly detection has been studied extensively by leveraging RGB information [47, 46, 14, 52, 42, 11, 62, 43] . Related works can be categorized into two branches: end-to-end and memorybased methods. Representative end-to-end methods exploit knowledge distillation [3, 62, 45] and normalizing flow [20, 57] to model the normal distribution. Instead, memory-based methods [44, 55] store normal features to construct normal prototypes. ZS 2D anomaly detection is proposed to target a challenging problem where training samples are inaccesible [17, 27, 33, 1, 16, 10] . WinCLIP [24] attempts to explore ZS 2D anomaly detection using CLIP. AnomalyCLIP [63] first introduces objectagnostic prompt learning to capture generic normality and abnormality, detecting anomalies across datasets. PromptAD [30] focuses on effectively fusing these embeddings to enhance zero-shot detection performance. Rendering quality PointAD interprets point clouds through their corresponding 2D renderings, and the quality of these renderings impacts the information that PointAD can extract from the original point clouds. In our manuscript, we used the Open3D Library to render the point clouds, but it does not provide an API for controlling rendering quality. To simulate varying rendering quality, we applied Gaussian blur with different extents σ to the 2D renderings. Sample visualizations are included in Figure 6 . Specifically, we conducted experiments on MVTec3D-AD using different blur σ values (i.e., ). Table 8 shows that the detection performance of PointAD diminishes as rendering quality decreases (with increasing sigma). However, the degradation is acceptable even when the renderings are heavily blurred (σ equals 9). In such cases, PointAD still outperforms baselines that use high-quality renderings.Input Resolution Here, we study the effect of resolutions of input point clouds. To create lowresolution point clouds, we downsample the entire high-resolution point clouds using Farthest Point Sampling (FPS) with various sampling ratios. This strategy allows us to generate corresponding low-resolution datasets for training and evaluating PointAD. Visualizations of these datasets are provided in Figure 7 . We train PointAD using the resulting low-resolution samples and test PointAD on the same resolution. Table 9 demonstrates that PointAD maintains a strong detection capacity for low-resolution point clouds when the downsampling ratios are 20%, 30%, 50%, and 70%. Even at 20% resolution, PointAD still achieves state-of-the-art performance. This indicates that PointAD is generally applicable to point clouds with various resolutions.Rendering angles and different numbers of views PointAD interprets point clouds from their 2D renderings, where rendering angles and numbers collectively determine the amount of information derived. They have different emphases. For rendering angles, the importance lies in the discrepancy between adjacent angles, as this affects the information granularity that PointAD obtains from adjacent views. When the angle discrepancy is fixed, the number of renderings determines the coverage of 3D information in the resulting 2D renderings. To capture all point cloud information, especially abnormal points, it is crucial to ensure comprehensive coverage. Therefore, our approach in selecting rendering angles and the number of renderings is to guarantee that all points in point clouds are adequately represented.Based on this principle, we conducted experiments to assess the impact of the number of renderings on PointAD's detection performance, circularly rendering point clouds to ensure even coverage of all points. As shown in Table 5 , increasing the number of views allows PointAD to gather more detailed information from the 2D renderings, benefiting from smaller angle discrepancies, which improves detection and localization results. However, when the number of views increased from 9 to 11, we observed a performance decline in PointAD, with the I-AUROC for global multimodal detection dropping from 87.4% to 86.4%. This suggests that incorporating too many views could introduce redundant information, resulting in 2D renderings with extensive overlap and excessive local detail. This overemphasis on local information can impede global recognition. Hence, the appropriate number of views benefits point understanding from informative views while mitigating the adverse effects of redundant local information. To further explore the impact of the absolute angle, we shift the rendering angles while keeping the angle discrepancy unchanged. The original adjacent angle discrepancy in our paper is 1 5 π. We divide this discrepancy into four parts and perform angle shifts of 1 20 π, 2 20 π, and 3 20 π to test the impact of varying rendering angles. Table 10 shows that PointAD maintains consistent performance across different rendering angles, demonstrating its robustness to variations in angles different from those used during training. Rendering lighting Further exploring the robustness of PointAD under different conditions could enhance its generalized detection performance. We conducted ablation studies to test its sensitivity under different lighting conditions and with occluded point clouds below. To evaluate the impact of rendering lighting, we adjusted the lighting conditions to render point clouds, generating variant datasets with different lighting. We used both stronger and weaker lighting to render point clouds compared to the original dataset, covering a broad lighting range. We denote stronger and the strongest lighting as "+" and "++", and weaker and the weakest lighting as "-" and "-". Visualizations of the resulting samples are presented in Figure 8 . The experiments were conducted on MVTec3D-AD, where we tested PointAD, trained on the original dataset, on these lighting variant datasets. Table 11 shows that PointAD can still detect anomalies even with significant discrepancies in rendering lighting, suggesting that PointAD is not sensitive to variations in rendering lighting.Point occlusions Next, we evaluated detection performance with occluded point clouds. We occluded point clouds by removing those invisible from a specific rendering angle and then used the same rendering parameters to project the remaining points. During this process, we observed that abnormal regions might be occluded totally. Unlike class classification, where class semantics remain unchanged by point occlusions, removing these anomaly semantics would transform anomaly samples into normal points. Therefore, we selected rendering angles that allow visibility of part or all anomalous regions when the point cloud is an abnormal instance. The occluded point clouds are shown in Figure 9 . We then used PointAD, trained on the original dataset, to test the resulting occluded dataset. Table 12 shows that PointAD suffers from performance degradation when the point clouds are occluded. We attribute this to two aspects: 1) Despite this strategy, the occluded In this section, we present the failure cases of PointAD, which we attribute to direct multimodality fusion. Since our model uses hybrid loss to incorporate the 3D and 2D anomaly semantics, it performs ZS multimodality 3D anomaly detection in a plug-and-play manner. However, when one modality prediction deviates severely from the ground truth in rare instances, direct fusion may result in an unpromising multimodal score map. As shown in Figure 10 , the hole in the cookie is visually similar to the chocolate on cookies, making it challenging to differentiate the hole anomaly via color information alone. Although PointAD can detect the hole based on its abnormal point relations, the RGB score map heavily influences the final multimodal score map. Conversely, the tire presents an inverse situation where RGB can effectively predict the anomalies, but the point score map fails to recognize it. The false detection could arise from unusual point density and distribution. To demonstrate the effect of point density, we randomly select normal regions of point clouds and subsequently increase or decrease the density of these regions through upsampling and downsampling. We provide a qualitative analysis in 11. The visualization shows that PointAD effectively resists noise at reasonable levels. However, when noise levels are extremely low or high, the corresponding regions become excessively sparse or dense. This causes normal regions to appear similar to hole anomalies or squeezed anomalies, leading PointAD to classify these noisy areas as anomalies. Non-parametric score alignment and filter methods could be a potential direction, which we leave for further work. Robustness to point density To investigate the impact of point density differences between the training and test datasets, we train PointAD using high-density point clouds (original dataset) and then test it on the low-density versions of the datasets, downsampled as described in Input resolution.Table 16 shows that PointAD can still detect anomalies even when retaining 50% of the points from the original point clouds. However, when more points are removed (30% and 20% sample ratio), PointAD experiences an obvious performance degradation. We attribute the misdetection to the overly sparse point clouds forming holes. Nevertheless, PointAD can still detect anomalies even withIn Table 17 , we provide a comparison of computation overhead among unsupervised and zero-shot manners 4 . The evaluation includes inference time per image, frames per second (FPS), and GPU memory consumption with a batch size of 1. For a fair comparison, we keep NVIDIA RTX 3090 24GB GPU free until we conduct experiments. Compared to PointCLIP V2, our model requires less time to infer an image, achieving higher FPS (2.52 vs. 0.66) with lower graphic memory usage (as discussed in Section 3.5). While CLIP + rendering has a slight advantage in computation overhead, our detection performance significantly outperforms it. Therefore, PointAD achieves a favorable trade-off between performance and computation overhead. To provide intuitive results, we compare the visualization of PointAD with PointCLIP V2. In Figure 12 , our model achieves accurate ZS 3D detection through the point cloud. Moreover, given RGB counterparts, PointAD further improves its detection capacity in M3D detection. However, PointCLIP V2 exhibits noisy activations for normal regions. After incorporating RGB information, PointCLIP V2 appears to struggle to fuse these two modalities in a plug-and-play manner, unlike PointAD.We respectively visualize the 2D renderings and corresponding 2D ground truths, which are rendered from 3D pint clouds and ground truths, as shown in Figure 13 . We also supplement more zero-shot segmentation results of PointAD in Figure 14 . Justification: we have created a separate "Limitations" section in our paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
 
 Appendix:  


Decision: The submissions got three positive recommendations. The reviewers liked this interesting idea for 3D anomaly detection, but also had some concerns on novelty, limitations, and effectiveness on more datasets. The authors addressed these concerns in the rebuttal and the comments during the authors and reviewers discussion period. The reviewers reached a consensus of acceptance, and did not have an intense discussion during the AC and the reviewers discussion period. The AC read through the paper, review, the confidential comments to the AC, rebuttal, and the discussion. The AC supports the reviewers’ decision, and makes a decision to accept this submission. This decision was approved by the SAC.

Reviews: review: paper_summary: paper_summary: The paper propose a unified framework, PointAD,to detect 3D anomalies in a ZS manner. Hybrid representation learning is proposed to incorporate the generic normality and abnormality semantics into PointAD. PointAD can incorporate 2D RGB information in a plug-and-play manner for testing, which can perform ZS M3D anomaly detection directly. Experiments results demonstrate the superiority of the model in detecting and segmenting 3D anomalies.
strength_weakness: strength_weakness: I do not notice any obvious weaknesses in this paper, but I still have a few questions that I hope the authors can answer.
1. The methodology, while innovative, is complex due to the need to render 3D objects into multiple 2D views and then re-project them, the performance of the model may heavily depends on the quality of the 2D renderings of the 3D objects, Therefore, if the renderings are of poor quality, could the model's performance degrade?
2. The experiments in the paper utilize high-resolution point cloud scans. Given that the point clouds obtained in practical applications may not always be high resolution, how effective is this method in detecting anomalies in low-resolution, sparse scans?
3. In the appendix's failure case section, the detection of the tire failed on the point score map. Thus, if a tire, or any object, has noise, how significantly does it affect the recognition on the point score map? I suspect that the possible reason for this failure could be the algorithm's sensitivity to point density and spatial distribution. When anomalies are located in areas where point density or distribution are inconsistent or significantly different from the training data, can this model still generalize well to new, unseen anomalies?
questions: questions: See weaknesses.
limitations: limitations: The authors adequately addressd the limitations. Societal impact is not discussed, but I believe it's not needed.
paper_summary: paper_summary: The paper introduces PointAD, the first approach to explore the domain of zero-shot 3D anomaly detection, leveraging CLIP's strong generalization to identify anomalies in previously unseen objects. It offers a unified framework that integrates 3D point clouds with 2D renderings, employing hybrid representation learning to capture the semantics of anomalies. PointAD's key contributions are its pioneering exploration of the ZS 3D anomaly detection domain, its ability to seamlessly integrate multimodal data like RGB information for enhanced detection, and its superior performance over existing methods. The robustness of the framework is confirmed through extensive experiments.
strength_weakness: strength_weakness: 1.	There’s some zero-shot anomaly detection methods with clip on 2D images, so this paper should compare with them, especially, from the view of techniques’ differences, not only the used data differences. For example, those zero-shot methods (WinCLIP, VAND and AnomalyCLIP) mentioned in [R1] and [R2].
[R1] Cao, Y., Xu, X., Zhang, J., Cheng, Y., Huang, X., Pang, G., & Shen, W. (2024). A survey on visual anomaly detection: Challenge, approach, and prospect. arxiv preprint arxiv:2401.16402.
[R2] Li, X., Huang, Z., Xue, F., & Zhou, Y. (2024). Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images. arxiv preprint arxiv:2401.16753.

2.	I also wonder if the new methods of 3D anomaly detection can also be used to deal with this zero-shot task. For example, [R3] and [R4]. 
[R3] Zuo, Z., Dong, J., Wu, Y., Qu, Y., & Wu, Z. (2024). CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation. arxiv preprint arxiv:2406.18941.
[R4] Li, W., Xu, X., Gu, Y., Zheng, B., Gao, S., & Wu, Y. (2024). Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22207-22216).
3.	More importantly, there’s some very similar studies with this work and task. Authors should mention and compare them within this study. For example, [R5].
[R5] Li, Y., Goodge, A., Liu, F., & Foo, C. S. (2024). Promptad: Zero-shot anomaly detection using text prompts. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1093-1102).
[R6] X. Chen, J. Zhang, G. Tian, H. He, W. Zhang, Y. Wang, C. Wang,Y. Wu, and Y. Liu, “Clip-ad: A language-guided staged dual-
path model for zero-shot anomaly detection,” arXiv preprint
arXiv:2311.00453, 2023.
[R7] Wang, C., Zhu, H., Peng, J., Wang, Y., Yi, R., Wu, Y., ... & Zhang, J. (2024). M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising. arxiv preprint arxiv:2406.02263.
4.	Computational Efficiency: Although the authors have commendably addressed computational challenges, the overall computational cost associated with rendering multi-view images and processing them through the vision encoder is still substantial. Since this field is still developing with few comparable studies, it is recommended that the authors provide a broader comparison about the computational cost with existing anomaly detection techniques, including those like Cheraghian, CPFM and 3DSR that are not strictly within the zero-shot category. This expanded comparison will help to show that the computational demands of PointAD are reasonable and suitable for practical use cases.
5.	Clarity and Completeness of Presentation: The paper's writing is generally clear and free of errors, yet minor aspects of the presentation could be polished for better reader comprehension. For instance, the figure overlays, specifically in Figure 1, currently only elucidate part (a), leaving explanations for parts (b) and (c) to be found in the main text. This separation can disrupt the reader's flow and understanding. Enhancing the figure overlays to include all relevant parts and ensuring that each component of the figure is self-explanatory will make the paper more accessible and its findings easier to grasp.
questions: questions: 1.	Please answer the main differences with these mentioned methods in Weaknesses and compare with part of them on experiments.
2.	Given the concerns raised in the Weaknesses section 1, does PointAD still hold the merit of computational efficiency when compared to other methods such as AnomalyCLIP and Cheraghian?
3.	Could you elaborate on the multi-view rendering process of point clouds (as described from line 134 to line 140), particularly how the color of the rendered images is determined? Furthermore, at line 254, it's stated that additional RGB information is utilized only in the test dataset. Given the description at line 564, which refers to 2D RGB information, how is this 2D data integrated with the rendered multi-view images? Are there any strategies which are employed to bridge the gap between the rendered training data and the RGB information used in testing?
limitations: limitations: The authors have discussed the limitations of their model, and there are no potential negative societal impacts associated with their work.
paper_summary: paper_summary: The paper titled "PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection" introduces PointAD, a novel framework designed for zero-shot 3D anomaly detection by leveraging both point clouds and RGB images. The approach builds on the strong generalization capabilities of CLIP and adapts it to 3D anomaly detection. The framework renders 3D anomalies into multiple 2D views and then projects these 2D representations back into 3D space. This hybrid representation learning allows for capturing generic anomaly semantics across various unseen 3D objects. PointAD also incorporates auxiliary point clouds to optimize learnable text prompts for better anomaly detection and segmentation.
strength_weakness: strength_weakness: 1. The sensitivity of the method to the selection of rendering angles and the number of views could be explored further.
2. While the experimental results are compelling, additional validation on more diverse and challenging datasets would further strengthen the claims.
3. The discussion of potential limitations and future work could be expanded to provide a more comprehensive view of the method's applicability and areas for improvement.
questions: questions: 1. Could the authors provide more insights into the selection of rendering angles and the impact of different numbers of views on the detection performance?
2. Have the authors considered the robustness of the method under different environmental conditions, such as varying lighting and occlusions?
3. Could additional experiments on other industrial or medical datasets help to further validate the generalizability of the proposed method?


======== post rebuttal ==========
The authors' rebuttal solve most of my concerns, hence I raise my score to borderline accept.
limitations: limitations: The authors have addressed some limitations, including the fixed rendering angles and the potential for further optimization of view selection. Constructive suggestions for improvement include exploring the sensitivity to rendering parameters and validating the method under different environmental conditions.

Comments: title: Response for Q2
Response for Q1
comment: We're glad to hear that your concerns have been addressed. If you have any further questions, please don’t hesitate to let us know. We are more than willing to assist with any remaining issues.
**Q2: Compare PointAD with those new methods on MVTec-3D AD**

 We find SOTA unsupervised methods (i.e., M3DM [1], BTF [2], and SDM [3]), which provide the codes. **We are pleased to compare these unsupervised methods as the upper boundary of performance.** Therefore, we follow their original settings, training category-specific models to report their results. Additionally, we also include them in the comparison of computation overhead. **These results will be incorporated into the revised version.**

|Methods|Point detection (Global)|Point detection (Local)|Multimodal detection (Global)|Multimodal detection (Local)|
|:---:|:---:|:---:|:---:|:---:|
|BTF (unsupevised)|76.3, 91.8|97.6, 92.3|89.8, 96.7|99.5, 97.1|
|SDM (unsupevised)|96.7, 90.9|97.0, 91.8|92.1, 97.6|93.3, 98.1|
|M3DM (unsupevised)|85.6, 93.4|92.8, 91.6|93.2, 92.7|98.4, 95.1|
|CPFM(unsupervised)|94.9, 98.7|97.6, 92.5|-,-|-,-|
|3DSR(unsupervised)|95.1, 94.3 |93.8, 91.2|97.3, 98.6|99.3, 97.4|
|WinCLIP (zero-shot)|45.2, 77.9|85.8, 59.4|38.7, 74.1|87.5, 64.2|
|VAND (zero-shot)|55.0, 83.4|86.4, -|68.6, 89.2|85.2, -|
|Ours (zero-shot)|82.0, 92.4|95.5, 84.4|86.9, 96.1|97.2, 90.2|

From Table 1, we can observe that PointAD even outperforms the unsupervised method on certain metrics. Specifically, PointAD outperforms BTF in global point detection, achieving 82.0 I-AUROC and 92.4 AP, compared to BTF's 76.3 I-AUROC and 91.8 AP. Besides, PointAD obtains higher P-AUROC for local point detection compared to M3DM, with scores of 95.5 vs. 92.8. It also surpasses SDM with a P-AUROC of 97.2 vs. 93.3 in the local multimodal detection. Although there is still room for improvement compared to these unsupervised methods, these results demonstrate the superiority of PointAD. 

|Methods|inference time (s)|FPS|GPU (MB)|Point detection (Global)|Point detection (Local)|Multimodal detection (Global)|Multimodal detection (Local)|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|BTF (unsupevised)|0.18|5.56|1934|76.3, 91.8|97.6, 92.3|89.8, 96.7|99.5, 97.1|
|SDM (unsupevised)|0.14|7.14|2716|96.7, 90.9|97.0, 91.8|92.1, 97.6|93.3, 98.1|
|M3DM (unsupevised)|2.86|0.35|7494|85.6, 93.4|92.8, 91.6|93.2, 92.7|98.4, 95.1|
|CPFM(unsupervised)|0.22 |4.55 |1379 |94.9, 98.7|97.6, 92.5|-,-|-,-|
|3DSR(unsupervised)|0.09 |11.11 |3067| 95.1, 94.3 |93.8, 91.2|97.3, 98.6|99.3, 97.4|
|WinCLIP (zero-shot)|0.29|3.45|3914|45.2, 77.9|85.8, 59.4|38.7, 74.1|87.5, 64.2|
|PointCLIP V2 (zero-shot)|1.52|0.66|9747|78.3, 49.4|87.4, 52.3|49.8, 79.3|51.2, 80.1|
|Cheraghian (zero-shot)|0.35 |2.86 |4847 | 53.6,81.7 | 88.2,57.0 |-,- | -,-|
|Ours (zero-shot)|0.40 |2.52 |4275|82.0, 92.4|95.5, 84.4|86.9, 96.1|97.2, 90.2|

Table 2 demonstrates that PointAD strikes a balance between computational overhead and performance. However, further reducing memory consumption remains a potential area for improvement.

If you have any further concerns or questions, please let us know. We are more than happy to address them.

**References:**

[1] Wang, Y., Peng, J., Zhang, J., Yi, R., Wang, Y., \& Wang, C. (2023). Multimodal industrial anomaly detection via hybrid fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8032-8041).

[2] Horwitz, E., \& Hoshen, Y. (2023). Back to the feature: classical 3d features are (almost) all you need for 3d anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2968-2977).

[3] Chu, Y. M., Liu, C., Hsieh, T. I., Chen, H. T., \& Liu, T. L. (2023, July). Shape-guided dual-memory learning for 3D anomaly detection. In Proceedings of the 40th International Conference on Machine Learning (pp. 6185-6194).
Thank you for pointing out your concerns. Please allow me to brief review the previous rebuttal. We have supplemented the performance comparison with WinCLIP and VAND in the previous response to **Q2**. As for the remaining methods, we do not compare them (i.e., PromptAD, CLIP3D-AD, IMRNet, and M3DM-NR) because they do not release their codes.

Next, we hope our responses could further help address your concerns.  

**Q1: The main innovations from this study**

The innovation of PointAD can be summarized in three key aspects: field contribution, main technological contribution, and applicability contribution.


**Field contribution:** Zero-shot 3D anomaly detection is a promising task, to which the conventional unsupervised methods are not visible, when the target 3D samples are unavailable due to privacy concerns or the absence of new products such as those generated by popular 3D generation fields. To the best of our knowledge, PointAD is the first to explore the challenging yet significant zero-shot task of detecting and localizing point anomalies across diverse objects and datasets. Our paper paves the way for future research in this unexplored field.


**Main technological contribution:** While CLIP demonstrates strong generalization capabilities for downstream tasks, its zero-shot performance for point anomaly detection and localization is limited (as shown in Tables 1 and 2 of the submitted manuscript). PointAD addresses this by proposing hybrid representation learning with tiny learnable parameters (only two text prompts), enabling CLIP to achieve accurate zero-shot 3D detection and zero-shot M3D detection.

The collaborative optimization innovatively captures generalized point anomaly semantics from both 3D and 2D perspectives. This enables PointAD to understand 3D anomalies through both point space and pixel space. Benefiting from it, PointAD could achieve zero-shot M3D in a plug-and-play manner, without any additional training.

**Applicability contribution:** PointAD transfers the extensive computation of point-level anomaly maps to a smaller feature space, significantly reducing memory consumption. This optimization improves PointAD's computational efficiency, allowing it to run on a single NVIDIA RTX 3090 24GB GPU, which enhances its applicability to real-world scenarios.
Thanks for the authors' response to my concerns. However, I found that you just compared yours with the new relevant papers I mentioned regarding technique differences. Ok, here I'll give two simple questions:

1) As you mentioned that "PointAD transfers the 2D generalization of CLIP to 3D point clouds and has the detection capacity for zero-shot 3D and M3D anomalies.". So, I wonder what your main innovations are from this study.

2) This study also used the MVTec-3D AD dataset, and this dataset has been cited with more than 40 references now. Why do you not compare yours with those new methods on this dataset? e.g., [1] with available codes; and others.


[1] Wang, Y., Peng, J., Zhang, J., Yi, R., Wang, Y., & Wang, C. (2023). Multimodal industrial anomaly detection via hybrid fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8032-8041).
The author's responses have addressed my concerns. I'm willing to raise my score to accept
Dear reviewers,

As the reviewer-author discussion period is about to end by 8.13, please take a look at other reviewers' reviews and authors' rebuttal at your earliest convenience. It would be great if you could ask authors for more clarification or explanation if some of your concerns are not addressed by the rebuttal.

Thanks,

AC
Your support is greatly appreciated. We're more than happy to take any further questions if otherwise.
Hi reviewers,

Thanks a bunch for all your hard work as reviewers for NeurIPS 2024.

The discussion period between reviewers and authors has started. Make sure to check out the authors' responses and ask any questions you have to help clarify things by 8.13.

Best,
AC