SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaqi Liu $^{1,*}$ , Guoyang Xie $^{1,2,*}$ , Ruitao Chen $^{1,*}$ , Xinpeng Li $^{1}$ , Jinbao Wang $^{1,\dagger}$ , Yong Liu $^{3}$ , Chengjie Wang $^{3,4}$ , Feng Zheng $^{1,\dagger}$
$^{1}$ Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
$^{2}$ NICE Group, University of Surrey, Guildford, GU2 7YX, United Kingdom
Tencent Youtu Lab, Shenzhen 518057, China
$^{4}$ Shanghai Jiao Tong University, Shanghai 200240, China
liujq32021@mail.sustech.edu.cn,guoyang.xie@ieee.org
linkingring@163.com, chenrt2022@mail.sustech.edu.cn
li.xin.peng@outlook.com,chaosliu@tencent.com
jasoncjwang@tencent.com, zfeng02@gmail.com
High-precision point cloud anomaly detection is the gold standard for identifying the defects of advancing machining and precision manufacturing. Despite some methodological advances in this area, the scarcity of datasets and the lack of a systematic benchmark hinder its development. We introduce Real3D-AD, a challenging high-precision point cloud anomaly detection dataset, addressing the limitations in the field. With 1,254 high-resolution 3D items (from forty thousand to millions of points for each item), Real3D-AD is the largest dataset for high-precision 3D industrial anomaly detection to date. Real3D-AD surpasses existing 3D anomaly detection datasets available regarding point cloud resolution (0.0010mm-0.0015mm), 360 degree coverage and perfect prototype. Additionally, we present a comprehensive benchmark for Real3D-AD, revealing the absence of baseline methods for high-precision point cloud anomaly detection. To address this, we propose Reg3D-AD, a registration-based 3D anomaly detection method incorporating a novel feature memory bank that preserves local and global representations. Extensive experiments on the Real3D-AD dataset highlight the effectiveness of Reg3D-AD. For reproducibility and accessibility, we provide the Real3D-AD dataset, benchmark source code, and Reg3D-AD on our website:https://github.com/M-3LAB/Real3D-AD.
Real 3D-AD Motivation:  $3\mathrm{D} > 2.5\mathrm{D}$ . There is a solid need to propose a high-resolution point cloud anomaly detection dataset to tap the gap between the academy and industry, which brings the capabilities of point cloud anomaly detection to the factory floor. Point cloud anomaly detection is widely deployed in real-world production lines. However, 3D anomaly detection datasets released in the academy are RGBD (2.5D), which is outside the demand of industrial manufacturing. Advanced machining and precision manufacturing require no blind spots throughout the inspection process. Nevertheless, the blind spots exist because RGBD datasets are achieved via single-view scanning. The lack of a real point cloud anomaly detection dataset hinders the further development of 3D
*Contributed Equally, † Corresponding Authors.
Candybar
Seahorse
Figure 1: Real3D-AD dataset examples for each category. The blue box indicates the normal images in the training dataset. The red box denotes the abnormal images in the test dataset. There are no blind spots in Real3D-AD since our dataset are achieved by scanning all the views of the object instead of the single view photoed by RGBD camera.
Toffees
anomaly detection. Because of this, it is crucial and urgent to propose a point cloud anomaly detection dataset that meets industrial manufacturing needs.
Limitation of current 3D-AD dataset and Real3D-AD advantage. To address this problem, we present a large-scale, high-resolution 3D anomaly detection dataset, Real3D-AD, to support the research and development of 3D anomaly detection methods. Although two 3D anomaly detection dataset (MVTec 3D-AD [20] and Eyescandies [3]) has been proposed, there are still several limitations: 1) the precision of MVTec 3D-AD are insufficient to satisfy the requirements of high-precision point cloud anomaly detection. Specifically, MVTec 3D-AD offers a limited number of 4,147 point clouds per object with a point precision of  $0.11\mathrm{mm}$ . Real3D-AD offers a significantly greater number of point clouds per object, estimated at 1.3 million, compared to MVTec 3D-AD, which is approximately 100 times larger. Moreover, the point precision of Real3D-AD reaches up to  $0.010\mathrm{mm}$ , which is ten times more advanced than MVTec 3D-AD. The detailed analysis is given in Table 1. 2) There are blind spots of 3D anomaly detection datasets if adopting an RGBD camera to collect the 3D data, like MVTec 3D and Eyescandies. Identifying defects may pose a challenge when relying solely on a single view for inspection. Real3D-AD is collected using high-resolution laser scans, perfect for spotting the product's defects everywhere, as shown in Figure 1. 3) The simulated dataset (Eyescandies) makes it hard to extend the realistic scenario. Due to the commercial privacy of real products, it is difficult to collect the CAD model of real-world products. So most researchers adopt the simulated software, like Blender framework [9]. Nevertheless, both synthesizing texture and anomaly details are not achieved with high fidelity. Real3D-AD collects the products from real-world applications and obtains excellent prototypes via the high-resolution 3D scanner. Hence, as shown in Table 3, we can conclude that three key characteristics distinguish Real3D-AD relative to prior work on the 3D anomaly detection dataset: high precision, no blind spot, and realistic high-accuracy prototype.
Benchmark & Baseline. To expedite research efforts towards developing a universal high-precision point cloud anomaly detection approach, we have constructed a comprehensive and structured large-scale benchmark, referred to ADBENCH-3D. Additionally, we have developed a registration-based baseline method that aligns with the prerequisites of high-resolution 3D anomaly detection. Given the practical constraints, the number of training datasets available for each category is restricted (less than or equal to four), as creating a high-accuracy prototype for each category is a time-intensive process (requiring up to two days per category). The configuration of ADBENCH-3D varies from contemporary unsupervised anomaly detection tasks in the realm of 3D. Section 4.1 provides a
comprehensive description of the setting. To be more specific, the training examples are limited ( $\leq 4$ ), and the test samples are only scanned by one side. The motivation is to simulate the real-world application: The scanning positions on the production line are fixed, and one position can only scan the results of one product side. Furthermore, to facilitate precise performance comparison within the community and guarantee replicability, the ADBENCH-3D framework encompasses a comprehensive end-to-end pipeline that includes data preprocessing, 3D-AD algorithms, evaluation scripts, metrics, and a visualization toolkit. ADBENCH-3D comprises a set of 8 fundamental 3D anomaly detection methodologies that have been implemented and tested on the Real3D-AD dataset. Moreover, the results presented in Table 4 indicate that the majority of current 3D-AD techniques are unable to attain satisfactory performance in Real3D-AD, as evidenced by their object-level AUROC score falling short of  $50\%$ . Consequently, we propose a registration-based 3D-AD method (Reg3D-AD) that serves as a versatile solution to cater to the requirements of the Real3D-AD dataset. The Rege3D-AD model introduces a novel feature memory bank, as illustrated in Figure 8, designed to preserve both the local and global features. The test objects are aligned with the training prototype during the inference process, and their features are extracted locally and globally. The defects are identified by assessing the distance between the features of the test object and the training prototypes. Therefore, Real3D-AD and ADBENCH-3D present a step towards unifying disjoint efforts in 3D anomaly detection research and pave the way toward a deeper understanding of 3D anomaly detection models.
- We create the first-ever high-resolution 3D anomaly detection dataset (Real3D-AD), enabling the design of high-resolution 3D anomaly detection algorithms and applying it to publicly available. Real3D-AD exhibits three primary attributes that set it apart from previous studies on 3D anomaly detection datasets. These attributes include a high level of precision, an absence of blind spots, and a realistic, high-accuracy prototype.
- The end-to-end pipeline offered by ADBENCH-3D includes data preparation, data splits, evaluation metrics and scripts, and visualization toolkits. ADBENCH-3D conducts a large-scale systematic assessment (8 main algorithms on Real3D-AD).
- We propose a general-purposed registration-based 3D anomaly detection method (Reg3D-AD). The efficacy of Reg3D-AD has been demonstrated through comprehensive experimentation on the Real3D-AD dataset, surpassing the performance of the subsequent most effective approach by a significant degree.
3D-AD Datasets. The datasets for 2D anomaly detection (2D-AD) are abundant, with a history tracing back to 2007 [29]. Over 20 different datasets are available for 2D-AD [11, 25, 17, 31]. The numerous 2D-AD datasets have given rise to many related works. Some studies have approached it from the perspective of image reconstruction [35, 8], feature distillation [12, 4, 27], and feature comparison [22]. A body of research also focuses on specific scenarios such as few-shot anomaly detection [30, 36, 7] and noisy anomaly detection [16]. In contrast, the number of datasets for 3D anomaly detection (3D-AD) is considerably limited. The first 3D-AD dataset was introduced in 2021, and today, there are only two 3D-AD datasets, MVTec 3D-AD dataset [20] and Eyecandies dataset [3]. MVTec 3D-AD [20] is a new dataset designed for 3D point cloud anomaly detection, and it is the only point cloud dataset for AD. It contains 2,656 pairs of images as training sets, 294 pairs of images as validation sets, 249 pairs of normal images, and 948 pairs of abnormal images to form a test set. The dataset comprises a total of 41 distinct types of anomalies, with a combined count of 1148 anomaly regions. Each pair of images consists of RGB images and tiff images representing the spatial coordinates of each pixel. The resolution of the images varies from  $400 \times 400$  to  $900 \times 900$ . The Eyecandies dataset [3] is a novel synthetic dataset comprising ten different categories of candies rendered in a controlled environment. It consists of 13,250 pairs of normal samples and 2,250 pairs of abnormal samples. Each depth image corresponds to six RGB images under different lighting conditions. Both MVTec 3D-AD and Eyecandies are RGBD datasets, limited to single-view information. To further explore the value of spatial information in the AD task, we propose the Real 3D-AD dataset, which expands the object information to 3D space. Prototypes in the Real 3D-AD training set encompass comprehensive object information from various views. The test set also includes multi-view information on objects, allowing for a more extensive exploration of the value of 3D information in AD tasks.
3D-AD methods. Recently, many high-quality papers have emerged in the field of 2D-AD [33, 37, 32, 26]. The release of MVTec 3D-AD also sparked interest in 3D-AD anomaly detection methods [15, 23, 5, 28, 8]. However, more research on 3D than 2D anomaly detection still needs to be done. Some methods only use depth information to remove background noise, which limits the use of depth information. Meanwhile, combining RGB and depth information without compromising performance remains a challenge. Bergmann et al. [1] propose a point cloud feature extraction network based on the teacher-student model. During training, the student and teacher networks maintain consistent features and use the differences in extracted features to locate anomalies during testing. Horwitz et al. [15] combine hand-crafted 3D descriptors with the classical AD approach KNN framework. While both of these methods are effective, their performance is poor. AST [23] performs well in MVTec 3D-AD but only uses depth information to remove background. AST still uses the 2D-AD method to detect anomalies, ignoring the depth information of the object. M3DM [28] extracts features from point clouds and RGB images separately and fuses them for better decision-making. This approach is superior to BTF but relies heavily on pre-trained large models and memory libraries. CPMF [6] also uses the KNN paradigm. However, it projects the point cloud into a two-dimensional image from different angles, significantly reducing feature extraction's complexity and computational cost and fusing the resulting information for detection. In summary, existing 3D-AD models either perform poorly or rely heavily on pre-trained models and memory libraries. Currently, there is a lack of anomaly detection methods that use point cloud information, and the available datasets for research in this field are only MVTec 3D-AD with depth information and artificially synthesized Eyescandies [3] datasets. To bring attention and research to this area, we introduce the Real3D-AD dataset.
3 Real3D-AD dataset
We outline the pipeline for generating the Real3D-AD dataset, including the description of a high-resolution scanner, the construction of prototypes, the generation of anomalies, and an assessment of the labor and time required for the process.
Figure 2: PMAX-S130.
Table 1: Comparison of data collection equipment.
Scanner	PMAX-S130	Zivid One-Plusa
Dataset	Real3D-AD (Ours)	MVTec 3D-AD [1]
FOV	100cm to 400cm	60cm to 200cm
Point Precision	0.011mm-0.015mm	0.11mm
Spatial Distance	0.04mm-0.07mm	0.37mm
3D Format	ASC, PLY, STL, OBJ, IGES	TIFF
${}^{a}$  https://www.zivid.com
Description of high-resolution and high-precision 3D scanner. To obtain precise 3D anomaly detection data, we utilize a high-resolution binocular 3D scanner called PMAX-S130, as illustrated in Figure 2. The PMAX-S130 optical system comprises a pair of lenses with low distortion properties, a high luminance LED, and a blue-ray filter. The blue light scanner has a lens filter that selectively allows only the blue light of a specific wavelength to pass through. The filter effectively screens most blue light due to its relatively low concentration in natural and artificial lighting. Nevertheless, using blue light-emitting light sources could pose a unique obstacle in this context. The image sensor can collect light using the lens aperture. Hence, the influence exerted by ambient light is vastly reduced. The device exhibits the ability to perform scanning operations under intricate lighting circumstances that are frequently encountered in workshop environments. The abovementioned objective is accomplished by employing a cold light source of high-brightness LEDs. This approach prolongs the device's longevity and reduces heat emissions while ensuring consistent scanning precision. Moreover, the precision of the device's scanning is augmented by integrating a lens with minimal distortion. The data presented in Table 1 demonstrates that the PMAX-S130 performs better than the Zivid camera (utilized by MVTec 3D-AD), particularly regarding point precision. Real3D-AD exhibits a higher point precision and spatial distance per cloud than MVTec 3D-AD, with a factor of 10 and 4.28, respectively. Thus, Real3D-AD provides a pathway to enhance comprehension in high-precision point cloud anomaly detection.
Figure 3: A prototype in the training set is made from two or more scan results.
Prototype construction. The prototype construction process is shown in Figure 3. Initially, the stationary object undergoes scanning while the turntable completes a full revolution of  $360^{\circ}$ , enabling the scanner to capture images of the various facets of the object. Subsequently, the object undergoes reversal, and the process of rotation and scanning is reiterated. Following the manual calibration of the front and back scanning outcomes, the algorithm precisely calibers the stitching process. If there are any gaps in the outcome, the scan stitching process is reiterated until the point cloud is rendered.
Figure 4: Anomalies annotation in Real3D-AD.
annotation(a)
normal sample(a)
normal sample(b)
abnormal sample(a)
abnormal sample(b)
Figure 5: Incompleteness annotation in Real3D-AD. The first column refers to the normal sample. The second column denotes the abnormal sample. The third column presents the annotation for the incompleteness. Specific, we mark the cross-section and the broken edges as anomalies, which do not introduce the extra points in the annotation.
annotation(b)
Anomalies types and labeling. The anomalies pertaining to point clouds can be classified into two categories: incompleteness and redundancy. CloudCompare (2016) [10] is utilized to annotate the point cloud data. The process of labeling is depicted in Figure 4. The first step involves importing a pcd file into the CloudCompare software and modifying the angle view and point cloud file size. Afterward, the anomalous and non-anomalous regions were segregated and assigned corresponding labels for each point cloud. The ultimate outcome is presented in a text file.
Labor and time-consuming. A significant requirement for labor characterizes the process of collecting and labeling the Real3D-AD dataset. Each prototypical construction requires 1.2 days
to complete each object with a team of three individuals. The initial individual assumes the task of conducting a scan, while the subsequent individual directs their attention toward manual calibration. The third individual is primarily responsible for the labeling aspect of the task. To generate anomalies, a team of four individuals is utilized to complete the task. The initial individual directs their attention towards the inadequacy of point cloud anomalies, while the subsequent individual assumes responsibility for the superfluousness of point cloud anomalies. The second individual is primarily concerned with the process of labeling anomalies. Each atypical specimen requires 5 hours to complete. Real3D-AD involves a team of seven individuals and a time frame of four months to complete, owing to the substantial workload involved.
3.2 Data Statistics
The statistical information of the Real3D-AD dataset is presented in Table 2. The table consists of the dataset category, the number of training prototypes, the number of normal and abnormal samples in the test dataset, the mean proportion of abnormal points in the test. Real3D-AD comprises a total of 1,254 samples that are distributed across 12 distinct categories. Each training set for a specific category contains only four samples, similar to the few-shot scenario in 2D anomaly detection. These categories include but are not limited to Airplane, Candybar, Chicken, Diamond, Duck, Fish, Gemstone, Seahorse, Shell, Starfish, and Toffees. All these categories are toys from manufacturing lines. The data presented in Table 2 demonstrates that the low anomaly point ratio poses a challenge for detecting anomalies. The majority of the attributes in the category pertain to transparency, indicating that the Real3D-AD dataset is highly suitable for tasks involving the detection of anomalies in point clouds.
Table 2: The statistics of Real3D-AD. Note that  $\Delta$  refers to the proportion of abnormal points in abnormal samples.
Category	Real Size [mm]	Attribute	Training	Test	Total	Anomaly Point Ratio Δ
Length	Width	Height	Normal	Normal	Abnormal
Airplane	34.0	14.2	31.7	Transparency	4	50	50	104	1.18%
Car	35.0	29.0	12.5	Transparency	4	50	50	104	1.99%
Candybar	33.0	20.0	8.0	Transparency	4	50	50	104	2.37%
Chicken	25.0	14.0	20.0	White	4	52	54	110	4.39%
Diamond	29.0	29.0	18.7	Transparency	4	50	50	104	5.41%
Duck	30.0	22.2	29.4	Transparency	4	50	50	104	2.00%
Fish	37.7	24.0	4.0	Transparency	4	50	50	104	2.86%
Gemstone	22.5	18.8	17.0	Transparency	4	50	50	104	2.06%
Seahorse	38.0	11.2	3.5	Transparency	4	50	50	104	4.57%
Shell	21.7	22.0	7.7	Transparency	4	52	48	104	2.25%
Starfish	27.4	27.4	4.8	Transparency	4	50	50	104	4.47%
Toffees	38.0	12.0	10.0	Transparency	4	50	50	104	2.46%
Mean	30.9	20.3	13.9	—	4	50	50	104	3.00%
Total	—	—	—	—	48	604	602	1254	—
Furthermore, a box-and-whisker plot represents the distribution of data points across all samples in the Real3D-AD dataset, as depicted in Figure 6. Two inferences can be made from the illustration in Figure 6. The observed differences in point count variability among distinct item categories within the point cloud demonstrate notable dissimilarity. In specific, the training samples provide complete prototypes of 3D objects while the test samples are scanned on only one side. Therefore, the number of training samples is much larger than that of test samples. Secondly, it can be observed that the disparity in point values between the normal and abnormal samples is comparatively minor within each test set.
3.3 Real3D-AD and other datasets
The findings presented in Table 3 demonstrate that Real3D-AD exhibits superior performance compared to MVTec 3D-AD [20] and Eyescandies [3], particularly in terms of point resolution, point precision, absence of blind spots, and dataset authenticity. Real3D-AD demonstrates a point resolution and precision of  $0.04\mathrm{mm}$  and  $0.011\mathrm{mm}$ , respectively. This is notably higher than MVTec 3D-AD, with a factor of 4.28 and 9 for point resolution and precision, respectively. Furthermore, the Real3D-AD system benefits from multi-view scanning, which eliminates any potential blind spots and thereby improves its anomaly detection capabilities. Therefore, Real3D-AD is deemed more suitable for achieving high levels of precision in point cloud anomaly detection and can accommodate industrial manufacturing requirements.
Figure 6: Point numbers for all samples on a logarithmic scale, visualized by a box-and-whisker plot.
Table 3: Comparison results of main datasets.
Dataset	MVTec 3D-AD	Eyescandies	Real 3D-AD (Ours)
Point Resolution	0.37mm	Not applicable	0.04mm
Point Precision	0.11mm	Not applicable	0.011mm
All Views (No Blind Spot)	X	X	✓
Real/Synthesis	Real	Synthesis	Real
4 Benchmark and Baseline
4.1 Problem Definition and Challenges
Problem definition. ADBENCH-3D setting can be formally stated as follows. Given a set of training examples  $\mathcal{T} = \{t_i\}_{i=1}^N$ , in which  $\{t_1, t_2, \dots, t_N\}$  are the training prototypes. In Real3D-AD, the number of prototypes for each category is limited  $(\leq 4)$ . In addition,  $\mathcal{T}_n$  belongs to one certain category,  $c_j \in \mathcal{C}$ , where  $\mathcal{C}$  denotes the set of all categories. During test time, given a normal or abnormal sample from a target category  $c_j$ , the AD model should predict whether or not the test 3D object is anomalous and localize the anomaly region if the prediction result is anomalous.
Training and test samples visualization. Figure 7 shows the training prototype and test dataset. The images in the blue box labeled (a)-(d) represent the prototype. The training prototype undergoes a  $360^{\circ}$  scan, ensuring no areas of limited visibility exist. The images in the orange box (e)-(h) represent the test sample. To simulate real-world conditions, the test sample is scanned on only one side. Since we desire to follow the real-world application: the workers or quality inspection equipments in the production line randomly check one side of the product to identify the defects by matching the scanned data with the prototype.
Challenges. The following are the three challenges. (1) Each category's training dataset contains only normal prototypes, i.e., no object or point-level annotations. (2) There are few normal prototypes of the training set available. There are fewer than four training prototypes in the setting of ADBENCH-3D. (3) There are unavoidable differences between the test set and the training set samples, which need to be addressed.
Figure 7: Examples of training and test samples in Real3D-AD.
4.2 ADBENCH-3D
Metrics. We standardize evaluation using metrics designed for 3D anomaly detection, including Area Under the Receiver Operating Characteristic Curve (AUROC) and the Area Under the