SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: ADBENCH-3D for Real3D-AD. The score indicates object-level AUROC  $\uparrow$ . The best results are highlighted in bold.
Category	BTF	M3DM	PatchCore	Reg3D-AD
Raw	FPFH	PointMAE	PointBERT	FPFH	FPFH+Raw	PointMAE
Airplane	0.730	0.520	0.434	0.407	0.882	0.848	0.726	0.716
Car	0.647	0.560	0.541	0.506	0.590	0.777	0.498	0.697
Candybar	0.539	0.630	0.552	0.562	0.541	0.570	0.663	0.685
Chicken	0.789	0.432	0.683	0.673	0.837	0.853	0.827	0.852
Diamond	0.707	0.545	0.602	0.627	0.574	0.784	0.783	0.900
Duck	0.691	0.784	0.433	0.466	0.546	0.628	0.489	0.584
Fish	0.602	0.549	0.540	0.556	0.675	0.837	0.630	0.915
Gemstone	0.686	0.648	0.644	0.617	0.370	0.359	0.374	0.417
Seahorse	0.596	0.779	0.495	0.494	0.505	0.767	0.539	0.762
Shell	0.396	0.754	0.694	0.577	0.589	0.663	0.501	0.583
Starfish	0.530	0.575	0.551	0.528	0.441	0.471	0.519	0.506
Toffees	0.703	0.462	0.450	0.442	0.565	0.626	0.585	0.827
Average	0.635	0.603	0.552	0.538	0.593	0.682	0.594	0.704
Precision-Recall curve (AUPR/AP). The details of metrics are introduced in the supplementary material.
Methods. As discussed in Section 2, 3D anomaly detection methods mainly focus on RGBD anomaly detection but not point anomaly detection tasks. So we adopt BTF [15] and M3DM [28] as our benchmark methods. We build a systematic benchmark, ADBENCH-3D, for our proposed Real3D-AD dataset, as shown in Table 4. In Table 4, BTF(Raw) refers to that we just adopt the coordinate features (xyz) into the BTF pipeline. BTF(FPFH) denotes we incorporate fast point feature histogram (FPFH) [24] into the BTF pipeline. M3DM(PointMAE) denotes M3DM using PointMAE [19] as a point cloud feature extractor and ignoring the RGB branch. M3DM(PointBERT) denotes M3DM using PointBert [34] as a point cloud feature extractor and ignoring the RGB branch. PatchCore+FPFH indicates we replace ResNet [14] as FPFH feature extractor and merge it into PathCore [21]. PatchCore+FPFH+Raw indicates we use FPFH and coordinates of each point cloud's feature and inject them into the PatchCore pipeline. PatchCore+PointMAE denotes that we adopt the PointMAE feature extractor and merge the feature into PatchCore architecture.
Toolkit. To complement ADBENCH-3D, we release a comprehensive toolkit as a starter code for high-precision point cloud anomaly detection, which implements 8 core methods in (1) data preprocessing, (2) evaluation scripts and metrics, and (3) visualization toolkit. Due to the page length limit, the toolkit details are put into the Github Repo.
4.3 Reg3D-AD
Inspired by PatchCore [22], we develop a general-purposed registration-based point cloud anomaly detection method (Reg3D-AD), as shown in Figure 8, which greatly meets the demands of Real3D-AD. Reg3D-AD utilizes a dual-feature representation approach to preserve the training prototypes' local and global features. Two distinct features are present in the dataset under consideration. The first feature pertains to the coordinate values of each point cloud, namely the x-, y-, and z-values. The second feature is the PointMAE feature, which characterizes the training prototypes in their entirety. The coordinate value encapsulates the localization attribute of individual points, whereas the PointMAE model prioritizes attaining a comprehensive representation of the training prototypes. The training phase aims to establish a repository of neighborhood-sensitive characteristics derived from all regular prototypes intended to serve as a memory bank. Before incorporating the novel functionalities into the memory repository, we implement the Coreset sampling technique to preserve the memory bank's size.
Anomaly score calculation. Before anomaly score calculation, the test 3D object needs to be registered via RANSAC algorithm [2]. After finishing the registration, the test 3D object is predicted as an anomaly if at least one point cloud is anomalous, and point-level anomaly segmentation is computed via the mean score of the point-level feature and global feature. In particular, with local feature bank  $\mathcal{M}^l$  and global feature bank  $\mathcal{M}^g$ , the object-level anomaly scores  $s$  for the test object  $x^{test}$  is computed by the mean value of local feature anomaly score  $s^l$  and the global feature anomaly score  $s^g$ . The local feature anomaly score is the maximum score  $s^{l*}$  between the test 3D object's
Figure 8: Pipeline of our baseline method. We extract features from the training set and sample the most representative features to the memory bank during training. During inference, we use the prototype as the target to calibrate the test sample and then extract the characteristics of the test sample to compare with the memory bank. We compute the anomaly score for each point according to the distance between test features and the memory bank.
point-level feature  $\mathcal{P}(x^{test})$  and its respective nearest neighbour  $m^{l*}$  in  $\mathcal{M}^l$ :
$$
m ^ {t e s t, *} = \underset {m ^ {t e s t} \in \mathcal {P} (x ^ {t e s t})} {\arg \max } \underset {m ^ {l} \in \mathcal {M} ^ {l}} {\min } \left\| m ^ {t e s t} - m ^ {l} \right\| _ {2}, \quad m ^ {l *} = \underset {m ^ {l} \in \mathcal {M} ^ {l}} {\arg \min } \left\| m ^ {t e s t} - m ^ {l} \right\| _ {2}, \tag {1}
$$
$$
s ^ {l *} = \left\| m ^ {\text {t e s t}, *} - m ^ {l *} \right\| _ {2}. \tag {2}
$$
To enhance the robustness of the anomaly detection model, PatchCore employs an important reweighting method [18] to tune the anomaly score:
$$
s ^ {l} = \left(1 - \frac {\exp \left\| m ^ {t e s t , *} - m ^ {*} \right\| _ {2}}{\sum_ {m \in \mathcal {N} _ {b} (m ^ {*})} \exp \left\| m ^ {t e s t , *} - m \right\| _ {2}}\right) \cdot s ^ {l *}, \tag {3}
$$
where  $\mathcal{N}_b(m^*)$  denotes  $b$  nearest patch-features in  $\mathcal{M}$  for test patch-feature  $m^*$ . The calculation of global feature anomaly scores  $s^g$  is similar to  $s^l$  and is achieved via the global feature memory bank  $M^g$ . Finally, the total anomaly score of each point cloud  $s^t = (s^l + s^g) / 2$ .
Analysis of ADBENCH-3D. The findings presented in Table 4 indicate that most of the 3D anomaly detection algorithms do not meet the requirements for Real3D-AD. Upon revisiting the setting outlined in Section 4.1, there appear to be notable similarities between this setting and that of few-shot anomaly detection. This is due to the fact that the training dataset for each category is comprised of a mere 4 prototypes. The majority of contemporary state-of-the-art 3D anomaly detection algorithms are not specifically designed for tasks involving the detection of anomalies in a few-shot scenario. To tackle this challenge, it is imperative to optimize the utilization of prototype data and guarantee that the acquired point cloud data remains unaffected by spatial relative positions. From Table 4, it can be clearly shown that our baseline method, Reg3D-AD, outperforms state-of-the-art 3D anomaly detection methods for the Real3D-AD dataset.
5 Limitations & Potential negative societal impacts
Limitations. There is still a broad scope for improvement and exploration based on our work. For example, our data is sourced from the 3D scanner and only contains spatial information, a common practice in industrial production. However, obtaining standardized RGB point cloud templates may be possible by calibrating and stitching multiple RGBD images or using modeling software rendering. RGB point cloud templates may be simultaneously applied to RGB image (2D) anomaly detection and point cloud (3D) anomaly detection. Additionally, our data can generate depth images from different angles by controlling rendering conditions, enabling anomaly detection from that perspective. This
has yet to be explored. Furthermore, although our baseline outperforms existing anomaly detection methods, it is still susceptible to false detection because the test point cloud edges are truncated. Therefore, more advanced models are expected to address these issues more effectively. Our work, as a first attempt at full-view point cloud anomaly detection, will inspire further exploration in this field.
Potential negative societal impacts. Our data is obtained from scanning industrial products, so no negative social impact will exist.
In this work, we propose a Real3D-AD dataset to investigate high-precision point cloud anomaly detection problems, which aims to facilitate the research of defect identification for advancing machining and precision manufacturing. The most extraordinary high-precision 3D industrial anomaly detection dataset to date, Real3D-AD, comprises 1,254 high-resolution 3D items ( $\geq$  one million point clouds for each item) spanning 12 objects of real-world applicability. Regarding point cloud resolution (0.0010mm-0.0015mm),  $360^{\circ}$  degree coverage, and flawless prototype, Real3D-AD outperforms currently available 3D anomaly detection datasets. In addition, we provide a thorough assessment of Real3D-AD datasets, highlighting the absence of baseline approaches to enable high-precision point cloud anomaly detection applications. We put forth a general registration-based 3D anomaly detection technique (Reg3D-AD) and 3D feature coupling unit that keeps local features and global representations. Experiments on the Real3D-AD dataset show that Reg3D-AD performs significantly better than the next-best approach.
Acknowledgments. This work is supported by the National Key R&D Program of China (Grant NO. 2022YFF1202903) and the National Natural Science Foundation of China (Grant NO. 62122035, 62206122).
[1] Paul Bergmann and David Sattlegger. Anomaly detection in 3d point clouds using deep geometric descriptors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2613-2623, 2023.
[2] Robert C Bolles and Martin A Fischler. A ransac-based approach to model fitting and its application to finding cylinders in range data. In *IJCAI*, volume 1981, pages 637-643, 1981.
[3] Luca Bonfiglioli, Marco Toschi, Davide Silvestri, Nicola Fioraio, and Daniele De Gregorio. The eyecandies dataset for unsupervised multimodal anomaly detection and localization. In Proceedings of the Asian Conference on Computer Vision, pages 3586-3602, 2022.
[4] Yunkang Cao, Qian Wan, Weiming Shen, and Liang Gao. Informative knowledge distillation for image anomaly segmentation. Knowledge-Based Systems, 248:108846, 2022.
[5] Yunkang Cao, Xiaohao Xu, Zhaoge Liu, and Weiming Shen. Collaborative discrepancy optimization for reliable image anomaly localization. IEEE Transactions on Industrial Informatics, 2023.
[6] Yunkang Cao, Xiaohao Xu, and Weiming Shen. Complementary pseudo multimodal feature for point cloud anomaly detection. arXiv preprint arXiv:2303.13194, 2023.
[7] Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen. Segment any anomaly without training via hybrid prompt regularization. arXiv preprint arXiv:2305.10724, 2023.
[8] Ruitao Chen, Guoyang Xie, Jiaqi Liu, Jinbao Wang, Ziqi Luo, Jinfan Wang, and Feng Zheng. Easynet: An easy network for 3d industrial anomaly detection. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7038-7046, 2023.
[9] Blender Online Community. Blender - a 3d modelling and rendering package. 2018.
[10] CloudCompare Community. Cloudcompare - a 3d pointcloud and mesh software. 2016.
[11] Yajie Cui, Zhaoxiang Liu, and Shiguo Lian. A survey on unsupervised industrial anomaly detection algorithms. arXiv preprint arXiv:2204.11161, 2022.
[30] Guoyang Xie, Jinbao Wang, Jiaqi Liu, Yaochu Jin, and Feng Zheng. Pushing the limits of fewshot anomaly detection in industry vision: Graphcore. In The Eleventh International Conference on Learning Representations, 2022.
[31] Guoyang Xie, Jinbao Wang, Jiaqi Liu, Jiayi Lyu, Yong Liu, Chengjie Wang, Feng Zheng, and Yaochu Jin. Im-iad: Industrial image anomaly detection benchmark in manufacturing. arXiv preprint arXiv:2301.13359, 2023.
[32] Xincheng Yao, Ruoqi Li, Jing Zhang, Jun Sun, and Chongyang Zhang. Explicit boundary guided semi-push-pull contrastive learning for supervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24490-24499, 2023.
[33] Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. Advances in Neural Information Processing Systems, 35:4571-4584, 2022.
[34] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[35] Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem-a discriminatively trained reconstruction embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8330-8339, 2021.
[36] Lingrui Zhang, Shuheng Zhang, Guoyang Xie, Jiaqi Liu, Hua Yan, Jinbao Wang, Feng Zheng, and Yaochu Jin. What makes a good data augmentation for few-shot unsupervised image anomaly detection? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4344-4353, 2023.
[37] Ying Zhao. Omnial: A unified cnn framework for unsupervised anomaly localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3924-3933, 2023.
A Experiment setup
Due to the limitation of GPU memory, we make subsampling on benchmark methods. For BTF [15], we sample and store the points of training samples at a ratio of 100:1, and for testing samples, we sample the points at a ratio of 500:1. After calculating the anomaly scores for each point, we estimate the anomaly scores for the unsampled points using the k-nearest neighbors (KNN) algorithm [13]. As for M3DM [28], we set the point groups number of point transformer to 16384. In our experiments based on PatchCore [22], we uniformly set the size of the memory bank to 10000.
Due to the page limit, we only give the object level auroc results in the main text, here we show the object level au-pr in Table 5, point level auroc in Table 6 and point level au-pr in Table 7. Among all the methods, our Reg3D-AD only slightly underperforms BTF (Raw) in terms of point-level AUROC, and slightly underperforms our self-designed PatchCore (FPFH+Raw) in terms of point-level AUPR. Overall, Reg3D-AD remains a reliable baseline. In addition, we adjusted different memory bank sizes and PointMAE point group numbers for comparison. As shown in Table 8, a larger PointMAE point group number improves model performance, while a larger memory bank does not always lead to performance improvements.
Table 5: ADBENCH-3D for Real3D-AD. The score indicates object-level AUPR ↑.
Category	BTF	M3DM	PatchCore	Reg3D-AD
Raw	FPFH	PointMAE	PointBERT	FPFH	FPFH+Raw	PointMAE
airplane	0.659	0.506	0.479	0.497	0.852	0.807	0.747	0.703
car	0.653	0.523	0.508	0.517	0.611	0.766	0.555	0.753
candybar	0.638	0.490	0.498	0.480	0.553	0.611	0.576	0.824
chicken	0.814	0.464	0.739	0.716	0.872	0.885	0.864	0.884
diamond	0.677	0.535	0.620	0.661	0.569	0.767	0.801	0.884
duck	0.620	0.760	0.533	0.569	0.506	0.560	0.488	0.588
fish	0.638	0.633	0.525	0.628	0.642	0.844	0.720	0.939
gemstone	0.603	0.598	0.663	0.628	0.411	0.411	0.444	0.454
seahorse	0.567	0.793	0.518	0.491	0.508	0.763	0.546	0.787
shell	0.434	0.751	0.616	0.638	0.573	0.553	0.590	0.646
starfish	0.557	0.579	0.573	0.573	0.491	0.473	0.561	0.491
toffees	0.505	0.700	0.593	0.569	0.506	0.559	0.708	0.721
Average	0.624	0.603	0.572	0.581	0.599	0.676	0.626	0.723
Table 6: ADBENCH-3D for Real3D-AD. The score indicates point-level AUROC ↑.
Category	BTF	M3DM	PatchCore	Reg3D-AD
Raw	FPFH	PointMAE	PointBERT	FPFH	FPFH+Raw	PointMAE
airplane	0.738	0.564	0.530	0.523	0.471	0.556	0.579	0.631
car	0.708	0.647	0.607	0.593	0.643	0.740	0.610	0.718
candybar	0.864	0.735	0.683	0.682	0.637	0.749	0.635	0.724
chicken	0.693	0.608	0.735	0.790	0.618	0.558	0.683	0.676
diamond	0.882	0.563	0.618	0.594	0.760	0.854	0.776	0.835
duck	0.875	0.601	0.678	0.668	0.430	0.658	0.439	0.503
fish	0.709	0.514	0.600	0.589	0.464	0.781	0.714	0.826
gemstone	0.891	0.597	0.654	0.646	0.830	0.539	0.514	0.545
seahorse	0.512	0.520	0.561	0.574	0.544	0.808	0.660	0.817
shell	0.571	0.489	0.748	0.732	0.596	0.753	0.725	0.811
starfish	0.501	0.392	0.555	0.563	0.522	0.613	0.641	0.617
toffees	0.815	0.623	0.679	0.677	0.411	0.549	0.727	0.759
Average	0.722	0.566	0.637	0.636	0.592	0.692	0.634	0.700
Table 7: ADBENCH-3D for Real3D-AD. The score indicates point-level AUPR  $\uparrow$
Category	BTF	M3DM	PatchCore	Reg3D-AD
Raw	FPFH	PointMAE	PointBERT	FPFH	FPFH+Raw	PointMAE
airplane	0.027	0.012	0.007	0.007	0.027	0.016	0.016	0.017
car	0.028	0.014	0.018	0.017	0.034	0.160	0.069	0.135
candybar	0.118	0.025	0.016	0.016	0.142	0.092	0.020	0.109
chicken	0.044	0.049	0.310	0.377	0.040	0.045	0.052	0.044
diamond	0.239	0.032	0.033	0.038	0.273	0.363	0.107	0.191
duck	0.068	0.020	0.011	0.011	0.055	0.034	0.008	0.010
fish	0.036	0.017	0.025	0.039	0.052	0.266	0.201	0.437
gemstone	0.075	0.014	0.018	0.017	0.093	0.066	0.008	0.016
seahorse	0.027	0.031	0.030	0.028	0.031	0.291	0.071	0.182
shell	0.018	0.011	0.022	0.021	0.031	0.049	0.043	0.065
starfish	0.034	0.017	0.040	0.040	0.037	0.035	0.046	0.039
toffees	0.055	0.016	0.021	0.018	0.040	0.055	0.055	0.067
Average	0.065	0.022	0.046	0.052	0.074	0.129	0.058	0.113
Table 8: Ablation experiment for our Reg3D-AD. We modified the point groups number of Point Transformer and the memory size of the memory bank. Note that O in O-AUROC refers to object-level, and P in P-AUROC refers to point-level.
Category	Ours(point groups 16384, memory size 10000)	Ours(point groups 8192, memory size 10000)	Ours(point groups 16384, memory size 20000)
O-AUROC	P-AUROC	O-AUPR	P-AUPR	O-AUROC	P-AUROC	O-AUPR	P-AUPR	O-AUROC	P-AUROC	O-AUPR	P-AUPR
airplane	0.716	0.631	0.703	0.017	0.873	0.704	0.819	0.024	0.737	0.622	0.761	0.014
car	0.697	0.718	0.753	0.135	0.752	0.777	0.744	0.183	0.719	0.735	0.706	0.170
candybar	0.827	0.724	0.824	0.109	0.693	0.751	0.709	0.077	0.732	0.704	0.744	0.070
chicken	0.852	0.676	0.884	0.044	0.621	0.457	0.628	0.040	0.648	0.689	0.636	0.052
diamond	0.900	0.835	0.884	0.191	0.794	0.792	0.794	0.297	0.882	0.817	0.877	0.122
duck	0.584	0.503	0.588	0.010	0.570	0.342	0.515	0.012	0.572	0.521	0.564	0.012
fish	0.915	0.826	0.939	0.437	0.836	0.818	0.848	0.325	0.948	0.825	0.953	0.418
gemstone	0.417	0.545	0.454	0.016	0.386	0.737	0.421	0.087	0.420	0.544	0.458	0.032
seahorse	0.762	0.817	0.787	0.182	0.713	0.773	0.702	0.191	0.726	0.811	0.707	0.202
shell	0.583	0.811	0.646	0.065	0.665	0.753	0.574	0.042	0.542	0.803	0.583	0.061
starfish	0.506	0.617	0.491	0.039	0.422	0.591	0.432	0.039	0.485	0.593	0.460	0.034
toffees	0.685	0.759	0.721	0.067	0.562	0.419	0.554	0.036	0.710	0.764	0.744	0.071
Average	0.705	0.700	0.723	0.113	0.666	0.681	0.653	0.120	0.674	0.697	0.677	0.108