SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiang Zhou $^{1*}$  Weize Li $^{1*†}$  Lihan Jiang $^{2*}$  † Guoliang Wang $^{1*}$  † Guyue Zhou $^{1}$  Shanghang Zhang $^{3}$  Hao Zhao $^{1}$
$^{1}$ AIR, Tsinghua University  $^{2}$ Wuhan University  $^{3}$ National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University {bamboosdu, liweize0224}@gmail.com shanghang@pku.edu.cn zhaohao@air.tsinghua.edu.cn
Object anomaly detection is an important problem in the field of machine vision and has seen remarkable progress recently. However, two significant challenges hinder its research and application. First, existing datasets lack comprehensive visual information from various pose angles. They usually have an unrealistic assumption that the anomaly-free training dataset is pose-aligned, and the testing samples have the same pose as the training data. However, in practice, anomaly may exist in any regions on a object, the training and query samples may have different poses, calling for the study on pose-agnostic anomaly detection. Second, the absence of a consensus on experimental protocols for pose-agnostic anomaly detection leads to unfair comparisons of different methods, hindering the research on pose-agnostic anomaly detection. To address these issues, we develop Multi-pose Anomaly Detection (MAD) dataset and Pose-agnostic Anomaly Detection (PAD) benchmark, which takes the first step to address the pose-agnostic anomaly detection problem. Specifically, we build MAD using 20 complex-shaped LEGO toys including 4K views with various poses, and high-quality and diverse 3D anomalies in both simulated and real environments. Additionally, we propose a novel method OmniposeAD, trained using MAD, specifically designed for pose-agnostic anomaly detection. Through comprehensive evaluations, we demonstrate the relevance of our dataset and method. Furthermore, we provide an open-source benchmark library, including dataset and baseline methods that cover 8 anomaly detection paradigms, to facilitate future research and application in this domain. Code, data, and models are publicly available at https://github.com/EricLee0224/PAD.
Unsupervised visual anomaly detection attempts to detect shape or texture anomalies using normal samples and representations from pre-trained model. Due to the increasing demand for quality control of manufactured products and the complicated shape of object and its texture information, object-centric anomaly detection has attracted growing interest. However, existing object anomaly detection datasets and methods are designed under the pose-aligned assumption, which limits the ability to detect potentially occluded anomalies from arbitrary pose views. By overcoming the inherent constraints of pose-aligned anomaly detection approaches, the pose-agnostic anomaly detection setting offers enhanced flexibility and applicability in real-world scenarios where a object may exhibit anomalies from diverse perspectives.
$\dagger$  Work done during internship at AIR.
Figure 1: (a) Existing assumption for Pose-aligned Anomaly Detection; (b) Our introduced Pose-agnostic Anomaly Detection setting.  $\mathbf{M}$  denotes pre-trained AD model.
To address the pose-agnostic anomaly detection problem, two key challenges need to be overcome. First, we currently lack datasets that simultaneously provide RGB images of object's multiple viewpoints and its pose labels. This limitation hinders the model's ability to learn the decision boundaries in high-level feature space of anomaly-free object in terms of both shape and texture, which is critical for performing accurate pose-agnostic anomaly detection. Second, in addition to above mentioned dataset being needed, how to measure and compare the performance of pose-agnostic anomaly detection models remains an issue. Existing benchmarks focus on pose-aligned anomaly detection, making it difficult to establish fair criterion for evaluating the effectiveness of pose-agnostic anomaly detection methods.
To this end, we developed the multi-pose anomaly detection (MAD) dataset that contributes to a comprehensive evaluation of pose-agnostic anomaly detection. Our dataset consists of 20 diverse LEGO toys, capturing more than 11,000 images from different viewpoints covering a wide range of poses. Furthermore, the dataset includes three types of anomalies that was carefully selected to reflect both the simulated and real-world environments. To explore our MAD dataset, we make in-depth analyses from two perspectives (shape and texture) with interesting observations. We conducted the analysis of the shape and texture complexity for different categories of LEGO toys in the dataset to gain insight into the performance differences of the baseline methods. By investigating the correlation between model performance and attribute complexity, we aim to assess whether the baseline methods are effective in capturing the full range of visual information from normal objects. Alongside the dataset, we propose OmniposeAD, which introduces neural radiance field into the anomaly detection paradigm for the first time. It overcomes the difficulty of learning the decision boundaries of a normal object from both shape and texture, has the ability to detect anomalies in multiple views of an object, and is even able to detect anomalies in poses that have never been seen in the training set. The main contributions are summarized as follows:
- We introduced Pose-agnostic Anomaly Detection (PAD), a challenging setting for object anomaly detection from diverse viewpoints/poses, breaking free from the constraints of stereotypical pose alignment, and taking a step forward to pratical anomaly detection and localization tasks.
- We developed the Multi-pose Anomaly Detection (MAD) dataset, the first attempt at evaluating the pose-agnostic anomaly detection. Our dataset comprises MAD-Sim and MAD-Real, containing 11,000+ high-resolution RGB images of multi-pose views from 20 diverse shapes and colors of LEGO toys, offering pixel-precise ground truth for 3 types of anomalies.
- We conducted a comprehensive benchmark of the PAD setting, utilizing the MAD dataset. Across 8 distinct paradigms, we meticulously selected 10 state-of-the-art methods for evaluation. In a pioneering effort, we delving into the previously unexplored realm of correlation between object attributes and anomaly detection performance.
- We proposed OmniposeAD, utilizing NeRF to encode anomaly-free objects from diverse viewpoints/poses and comparing reconstructed normal reference with query image for pose-agnostic anomaly detection and localization. It outperforms previous methods quantitatively and qualitatively on the MAD benchmark, which indicates the promises of PAD setting.
Table 1: Comparison of MAD and existing object anomaly detection datasets. Abbreviated letters refer to the details<sup>1</sup>.
Datasets	Year.	Type.	Represent.	Sample statistics	Object Attributes
#Cls.	#Normal	#Abnormal	Color	Structure	Pose
GDXray[25]	2015	Real	RGB-D	1	0	19,407	Grayscale	C	X
MVTec:
AD[5]	2019	Real	RGB	15	4,096	1,258	Diverse	S	X
3D AD[7]	2021	Real	RGB/PC	15	2,904	948	Diverse	S	X
LOCO-AD[4]	2022	Real	RGB	5	2,347	993	Diverse	C	X
MPDD[20]	2021	Real	RGB	6	1,064	282	Diverse	C	X
Eyecandies[8]	2022	Syn.	RGB/D/N	10	13,250	2,250	Diverse	S	X
VisA[65]	2022	Real	RGB	12	9,621	1,200	Diverse	S&C	X
MAD (Ours)	2023	Sim+Real	RGB	20	5,231	4,902	Diverse	S&C	✓
Object Anomaly Detection Datasets. The progress of object anomaly detection in industrial vision is significantly impeded by the scarcity of datasets containing high-quality annotated anomaly samples and comprehensive view information about normal objects. MVtec has developed a series of widely-used photo-realistic industrial anomaly detection datasets: The objects provided by the AD[5] dataset are simple, as discerning anomalies can be achieved solely from a single view. Although the 3D-AD[7] dataset offers more complex objects, it lacks RGB information from a full range of views, requiring the supplementation of hard-to-capture point cloud data to detect subtle structural anomalies. The LOCO AD[4] dataset provides rich global structural and logical information but is not suitable for fine-grained anomaly detection on individual objects. GDXray[25] provides grayscale maps obtained through X-ray scans for visual discrimination of structural defects but lacks normal samples and color/texture information. The MPDD[20] dataset offers multi-angle information about the objects but is limited in size and lacks standardized backgrounds in the photos. Recently, Eyecandies[8] has introduced a substantial collection of synthetic candy views captured under various lighting conditions and provides multi sensory inputs. However, there remains a significant gap between synthesized data and the real or simulated data domain. VisA[65] contains objects with complex structures, multiple instances at different locations in single view, different objects across 3 domains, and multiple anomaly classes, thus is closer to the real world than MVtec-AD[5]. To address these issues and enable exploration of the pose-agnostic AD problem, we propose Multi-pose Anomaly Detection (MAD) dataset. We present comprehensive comparison between MAD and other representative object anomaly detection datasets in Table.1.
Unsupervised Anomaly Detection for Visual Inspection. Existing methods can be categorized into feature embedding-based and reconstruction-based strategies, according to a recent survey[24]. Feature embedding-based methods such as Teacher-Student Architecture[6, 51, 15, 10, 35], One-Class Classification[17, 56, 62, 31, 22, 54], Distribution-map[36, 42, 33, 34, 18, 59, 44], and Memory Bank[32, 13, 21, 2, 50] aim to learn low-dimensional representations of input data to capture underlying patterns and anomalies. Reconstruction-based methods exploiting Autoencoder (AE)[3, 53, 61, 60, 14, 46], Generative Adversarial Networks (GANs)[38, 37, 52, 39, 23], and Transformer[27, 58, 29, 57] model aim to learn a representation that can effectively reconstruct normal samples. Deviations in reconstruction quality indicate the presence of anomalies. BTF[19] reveals that using only angularly limited RGB images to determine whether an object is defective or not can easily overlook some obscured structural anomalies. Therefore it is necessary to develop methods for 3D anomaly detection using 3D information, such as point clouds, depth, etc. AST[35] employs RGB image with depth information to enhance anomaly detection performance. M3DM[47] and CPMF[11] encourage the fusion of features from different modalities in RGB and point cloud information, and combine the local geometric information of the 3D modal with the global semantic information of the pseudo 2D modal.
Neural Radiance Field. Neural Radiation Field (NeRF)[26] implements an implicit representation of the scene, receiving hundreds of sampling points along each camera ray and outputting the predicted color and density. Because NeRF can handle complex illumination and occlusion situations, we can use it to capture the color and shape properties of objects in real scenes, which means that NeRF has
<sup>1</sup>Syn., PC, D, N, C and S denotes Synthesis, Point Cloud, Depth, Normal Map, Complex, and Simple, respectively.
the potential to be applied to anomaly detection tasks. It has also been used for numerous applications, such as large-scale urban reconstruction[48, 49, 40], human face or body reconstruction[55, 12], and robotic applications such as pose estimation and SLAM[41, 64, 63]. Since NeRF allows for the continuous modeling of 3D scenes and the re-rendering of high-quality photos, it can create reference images from any angle for anomaly detection in this work.
3 MAD: Multi-pose Anomaly Detection Dataset
Figure 2: (a) Our LEGO family in MAD dataset; (b) Examples of three defect types.
3.1 Data Selection
Aiming to support pose-agnostic anomaly detection, our dataset selects 20 LEGO animals toys with diverse shape complexity and color contrasts as target objects shown in Figure.2(a). These selected toys encompass a wide range of shapes and sizes, ranging from cat and pig to larger animals like unicorn and elephant, along with other interesting animal forms. Each animal toy is meticulously designed and constructed to ensure a variety of shapes and complexities. In selecting these objects, we also paid attention to color contrasts. Bright and vibrant colors are used in the construction of each animal toy, making them more visually striking and easily recognizable in the images. The choice of color contrasts aims to assist algorithms in accurately detecting and localizing anomalies in different environments and backgrounds.
3.2 Data Processing and Annotation
We used Blender in combination with Ldrew (LEGO parts library) to build 20 different types of 3D LEGO animal models and generate anomalous such as burrs, stains and missing parts shown as Figure.2(b). To achieve a full pose angle rendering of the normal samples, we established the center of the animal as the origin and positioned the rendering camera around the sphere. By maintaining a fixed radius, we generated multiple cameras simultaneously, each positioned at a different theta/phi coordinate in the spherical coordinate system. In addition, we created ten cameras equally spaced along the Z-axis, with the camera orientation set to face the animal. In order to edit the real anomalies on the simulation model, we hired professional LEGO players to manually modify the 20 types of LEGO animal models according to the types of anomalies encountered using PhotoShop. For the hand-edited anomalies, synchronized ground truth labels were produced by a computer vision researcher. Notably, only a subset of images showed the anomaly portion, which required a manual selection process to determine the appropriate data for anomaly detection. Ultimately, approximately 150-300 anomaly images were generated for each animal. To generate training data with more normal instances, we used the same approach. For the real dataset, we collect parts from the actual production line, including abnormal parts characterized by real burrs and stains. In addition, we manually generated the missing-piece anomaly by removing LEGO parts. Given the rarity of anomalies in the manufacturing process of LEGO toys and the difficulty in obtaining these parts, we deliberately reduced the sample size of the real-world version.
3.3 Data Statistics and Split
We present the statistical information of the dataset in Table.1. During the data splitting process, we followed design strategy of existing anomaly detection datasets. MAD-Sim is a dataset created programmatically using Blender, allowing for controlled sample quantities. We divided it into training and test sets with a sample ratio of approximately 2:1 to explore anomaly detection algorithm performance on multi-view objects. For MAD-Real, the data was captured in real-world scenes using a camera. Anomalies are infrequent in actual production, and we aimed to reflect this reality. MAD-Real has approximately one-fifth the total sample count of MAD-Sim and follows a similar division into training and test sets, maintaining a sample ratio of about 4:1.
3.4 Object Attributes Quantification
Anomaly detection algorithms are evolving rapidly, but its often difficult to intuitively determine which one is the best from performance metrics when trying to deploy an algorithm to a real scenario. To reduce testing costs, we explore a novel aspect in anomaly detection tasks, which is the relationship between object attributes and anomaly detection performance. When we are given an object to be tested, we expect to determine which algorithm is likely to be the best, simply by referring to the correlation between quantitative metrics of the object's attributes, i.e., object shape complexity and color contrast, and the performance of different algorithms. The quantitative details are shown in Fig.3.
Figure 3: Object attributes quantification of 20 LEGO toys.
4 Pose-agnostic Anomaly Detection Framework
Our Pose-agnostic Anomaly Detection (PAD) setting introduced for object anomaly detection and localization tasks is shown as Figure.1 and it can be formally stated as follow: Given a training set  $\mathcal{T} = \{t_i\}_{i=1}^N$ , in which  $\{t_1, t_2, \dots, t_N\}$  are the anomaly-free samples from object's multi pose view and each sample  $t$  consists of a RGB image  $I_{\mathrm{rgb}}$  w/ pose information  $\theta_{\mathrm{pose}}$ . In addition,  $\mathcal{T}$  belongs to certain object  $o_j$ ,  $o_j \in \mathcal{O}$ , where  $\mathcal{O}$  denotes the set of all objects categories. During testing, given a query (normal or abnormal) image  $\mathcal{Q}$  from object  $o_j$  w/o pose information  $\theta_{\mathrm{pose}}$ , the pre-trained AD model  $M$  should discriminate whether or not the query image  $\mathcal{Q}$  is anomalous and localize the pixel-wise anomaly region if the anomaly is detected.
4.2 OmniposeAD
The OmniposeAD illustrated in Figure.4, consists of anomaly-free neural radiance field, coarse-to-fine pose estimation module, and anomaly detection and localization module. The input of OmniposeAD is query image  $\mathcal{Q}$  w/o pose. Initially,  $\mathcal{Q}$  undergoes the coarse-to-fine pose estimation module to obtain the accurate camera view pose  $\hat{\theta}_{\mathrm{pose}}$ . Subsequently,  $\hat{\theta}_{\mathrm{pose}}$  is utilized in the pre-trained neural radiance field for rendering the normal reference. Finally, the reference is compared to the  $\mathcal{Q}$  to extract the anomaly information.
4.2.1 Anomaly-free reference view synthesis
To reconstruct anomaly-free references from various viewpoints, we employ unsupervised algorithm capable of synthesizing novel views. With an anomaly-free training set  $\mathcal{T}$ , we represent anomaly-free
Figure 4: OmniposeAD consists of three components. 1) The Neural Radiance Field is trained using multi-pose views of anomaly-free objects to capture normal information. 2) The query image proceeds through two-stage pose estimation process, from coarse to fine: a matching descriptor is generated to match with the anomaly-free database samples to obtain coarse pose, then the pose refined using the iNeRF[55] to obtain the reference pose. 3) The pre-trained Neural Radiance Field renders the normal reference, which is compared with the query image for anomaly detection and localization.
objects implicitly using vanilla neural radiance field, following [26] which employs a multi layer perceptron (MLP) network and minimizes the photometric loss.
4.2.2 Coarse-to-fine pose estimation
To generate normal reference using the pre-trained neural radiance field, the fine camera pose  $\hat{\theta}_{\mathrm{pose}}$  of the query image needs to be determined. For this purpose, we employ the coarse-to-fine pose estimation module that combines image retrieval techniques and iNeRF[55].
On a coarse scale Our objective is to roughly align the query image  $\mathcal{Q}$  with an anomaly-free image with camera pose in the training database i.e., find a function  $\phi$  that satisfies the following criteria:
$$
\phi : \mathbb {R} ^ {H \times W \times 3} \mapsto \mathbb {R} ^ {6 4} \quad \phi (\mathcal {Q}) = \mathbf {q} \in \mathbb {R} ^ {6 4} \tag {1}
$$
First, we utilize the pre-trained EfficientNet-B4 model[43] as the backbone for feature extraction. Following the standard procedure[58], we input the multi-scale feature map into the network to obtain the descriptor q. Given an 3-channel RGB query image  $I_{\mathrm{rgb}}$  of dimension  $H \times W$ , our model  $\phi$  extract a compact 64-dimensional (64D) descriptor or embedding  $\phi(x)$ . Then the image retrieval task[9, 1, 30] considers a database of images  $I_{\mathrm{rgb}}$  for each sample  $t_i$  in the training set  $\mathcal{T}$ , and given query image  $Q$ , we calculate:
$$
\underset {\mathcal {T}} {\operatorname {a r g m i n}} \| \phi (\mathcal {Q}) - \phi (I) \| _ {2} ^ {2} \tag {2}
$$
Finally, the top-1 most similar image  $I_{\mathrm{rgb}}$  is retrieved. We utilize the associated pose  $\theta_{\mathrm{pose}}$  of the image  $I_{\mathrm{rgb}}$  as the initial coarse pose  $\hat{\theta}_{\mathrm{pose}}$  for the next stage.
On a fine scale We refine the estimated pose using iNeRF[55]. This method enables us to fully utilize the pre-trained NeRF model and estimate camera poses. iNeRF uses the ability from NeRF to take some estimated camera pose  $\theta \in \mathrm{SE}(3)$  in the coordinate frame of the NeRF model and render a corresponding image observation, and then use the same photometric loss function as NeRF but updates the fine pose  $\hat{\theta}_{\mathrm{pose}}$  instead of the weights of the MLP network.
4.2.3 Anomaly detection and localization
When the normal reference is rendered using  $\hat{\theta}_{\mathrm{pose}}$  and query image  $\mathcal{Q}$  are available, we use the frozen pre-trained CNN backbone for feature extraction. The feature from layer1 to layer5 are resized
to the same size as the query image. Then the multi-scale feature and the query image are concatenate together to form a feature map,  $q \in R^{C \times H \times W}$  with the corresponding input pixel. The query image corresponds to  $f_{q} \in R^{C \times H \times W}$ , while the rendered reference corresponds to  $\hat{f} \in R^{C \times H \times W}$ . We first define the feature difference map,  $d(i, u)$ , as shown in Eqn. 3
$$
\boldsymbol {d} (i, u) = \boldsymbol {f} _ {\boldsymbol {q}} (i, u) - \hat {\boldsymbol {f}} (i, u), \tag {3}
$$
where  $i$  denotes the channel index and,  $u$  is the spatial position index (height together with width for simplicity).
$$
\boldsymbol {s} (u) = \left| \left| \boldsymbol {d} (:, u) \right| \right| _ {2}. \tag {4}
$$
Anomaly detection aims to determine if an image contains anomalous regions. We intuitively use the maximum value of the averagely pooled  $s(u)$  as the anomaly score of the whole image.
Anomaly localization aims to localize anomalous regions, producing an anomaly score map,  $s(u)$  in Eqn. 4, which assigns an anomaly score for each pixel,  $u$ .  $s(u)$  is calculated as the  $L2$  norm of the feature difference vector,  $d(:, u)$  in Eqn. 3.