SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Quantitative results of pixel/image-level anomaly detection performance on MAD.
Category	Patchcore[32]	STFPM[45]	Feature Embedding-based	Reconstruction-based	Ours
Fastflow[59]	CFlow[18]	CFA[21]	Cutpaste[22]	DRAEM[16]	FAVAAE[51]	OCRGAN[28]	UniAD[57]
Gorilla	88.4/66.8	93.8/65.3	91.4/51.1	94.7/69.2	91.4/41.8	36.1/-	77.7/58.9	92.1/46.8	94.2/-	93.4/56.6	99.5/93.6
Unicorn	58.9/92.4	89.3/79.6	77.9/45.0	89.9/82.3	85.2/85.6	69.6/-	26.0/70.4	88.0/68.3	86.7/-	86.8/73.0	98.2/94.0
Mallard	66.1/59.3	86.0/42.2	85.0/72.1	87.3/74.9	83.7/36.6	40.9/-	47.8/34.5	85.3/33.6	88.9/-	85.4/70.0	97.4/84.7
Turtle	77.5/87.0	91.0/64.4	83.9/67.7	90.2/51.0	88.7/58.3	77.2/-	45.3/18.4	89.9/82.8	76.7/-	88.9/50.2	99.1/95.6
Whale	60.9/86.0	88.6/64.1	86.5/53.2	89.2/57.0	87.9/77.7	66.8/-	55.9/65.8	90.1/62.5	89.4/-	90.7/75.5	98.3/82.5
Bird	88.6/82.9	90.6/52.4	90.4/76.5	91.8/75.6	92.2/78.4	71.7/-	60.3/69.1	91.6/73.3	99.1/-	91.1/74.7	95.7/92.4
Owl	86.3/72.9	91.8/72.7	90.7/58.2	94.6/76.5	93.9/74.0	51.9/-	78.9/67.2	96.7/62.5	90.1/-	92.8/65.3	99.4/88.2
Sabertooth	69.4/76.6	89.3/56.0	88.7/70.5	93.3/71.3	88.0/64.2	71.2/-	26.2/68.6	94.5/82.4	91.7/-	90.3/61.2	98.5/95.7
Swan	73.5/75.2	90.8/53.6	89.5/63.9	93.1/67.4	95.0/66.7	57.2/-	75.9/59.7	87.4/50.6	72.2/-	90.6/57.5	98.8/86.5
Sheep	79.9/89.4	93.2/56.5	91.0/71.4	94.3/80.9	94.1/86.5	67.2/-	70.5/59.5	94.3/74.9	98.9/-	92.9/70.4	97.7/90.1
Pig	83.5/85.7	94.2/50.6	93.6/59.6	97.1/72.1	95.6/66.7	52.3/-	65.6/64.4	92.2/52.5	93.6/-	94.8/54.6	97.7/88.3
Zalika	64.9/68.2	86.2/53.7	84.6/54.9	89.4/66.9	87.7/52.1	43.5/-	66.6/51.7	86.4/34.6	94.4/-	86.7/50.5	99.1/88.2
Phoenix	62.4/71.4	86.1/56.7	85.7/53.4	87.3/64.4	87.0/65.9	53.1/-	38.7/53.1	92.4/65.2	86.8/-	84.7/55.4	99.4/82.3
Elephant	56.2/78.6	76.8/61.7	76.8/61.6	72.4/70.1	77.8/71.7	56.9/-	55.9/62.5	72.0/49.1	91.7/-	70.7/59.3	99.0/92.5
Parrot	70.7/78.0	84.0/61.1	84.0/53.4	86.8/67.9	83.7/69.8	55.4/-	34.4/62.3	87.7/46.1	66.5/-	85.6/53.4	99.5/97.0
Cat	85.6/78.7	93.7/52.2	93.7/51.3	94.7/65.8	95.0/68.2	58.3/-	79.4/61.3	94.0/53.2	91.3/-	93.8/53.1	97.7/84.9
Scorpion	79.9/82.1	90.7/68.9	74.3/51.9	91.9/79.5	92.2/91.4	71.2/-	79.7/83.7	88.4/66.9	97.6/-	92.2/69.5	95.9/91.5
Obesobeso	91.9/89.5	94.2/60.8	92.9/67.6	95.8/80.0	96.2/80.6	73.3/-	89.2/73.9	92.7/58.2	98.5/-	93.6/67.7	98.0/97.1
Bear	79.5/84.2	90.6/60.7	85.0/72.9	92.2/81.4	90.7/78.7	68.8/-	39.2/76.1	90.1/52.8	83.1/-	90.9/65.1	99.3/98.8
Puppy	73.3/65.6	84.9/56.7	80.3/59.5	89.6/71.4	82.3/53.7	43.2/-	45.8/57.4	85.6/43.5	78.9/-	87.1/55.6	98.8/93.5
Mean	74.7/78.5	89.3/59.5	86.1/60.8	90.8/71.3	89.8/68.2	59.3/-	58.0/60.9	89.4/58.0	88.5/-	89.1/62.2	97.8/90.9
Our experiments aim to broadly evaluate the performance of anomaly detection algorithms in a pose-agnostic setting, where different types of anomalies may appear in different object poses. We selected representative state-of-the-art methods from different paradigms for fair preliminary benchmarking using the MAD dataset we developed with the aim of exploring the applicability of existing methods in the PAD setting. In addition, we evaluate the performance of our proposed OmniposeAD, normal reference reconstruction results, and the effect of training poses/viewpoints richness on performance.
Benchmark Methods Selection. The selection criteria for benchmark methods include representativeness, superior performance, and availability of source code. To comprehensively investigate the performance of anomaly detection algorithms in the pose-agnostic anomaly detection setting, we selected representative methods from 8 paradigms, the taxonomy can be found in the supplementary. For the Feature Embedding-based strategy, we selected Patchcore[32], STFPM[45], Fastflow[59], CFlow[18], CFA[21], and Cutpaste[22]. For the Reconstruction-based strategy, we selected DRAEM[16], FAVAE[51], OCR-GAN[28], and UniAD[57]. It is important to note that 3D anomaly detection methods utilizing multimodal information were not considered in order to ensure a fair comparison, as the MAD dataset only provides RGB information.
Implementation Details. For the implementation details, we trained the OmniposeAD using the MAD dataset, employing the same hyperparameters for both synthetic and real scenes. During the anomaly-free novel view synthesis phase, the process took approximately 5 hours with a batch size of 4096. The image resolution was set to  $400 \times 400$  for the simulated dataset and  $504 \times 378$  for the
real dataset to ensure efficient processing. Additionally, the coarse-to-fine pose estimation process involved a total of 300 optimization steps. We initialized the learning rate to 0.01 and decayed it to 5.8e-05 at epoch 50 to facilitate a smooth optimization process, utilizing a batch size of 3072. The entire framework required approximately 10-15 hours to run on a single NVIDIA Tesla A100.
Evaluation Metric. Following previous work, we specifically choose the Area Under the Receiver Operating Characteristic Curve (AUROC) as the primary metric for evaluating the performance of anomaly segmentation at the pixel-level and anomaly classification at the image-level. While there exist various evaluation metric for these tasks, AUROC stands out as the most widely used and suitable metric for conducting comprehensive benchmarking. The AUROC score can be calculated as follows:
$$
\mathrm {A U R O C} = \int \left(\mathrm {R} _ {\mathrm {T P}}\right) \mathrm {d} \mathrm {R} _ {\mathrm {F P}} \tag {5}
$$
Here,  $\mathrm{R_{TP}}$  and  $\mathrm{R_{FP}}$  represent the pixel/image-level true positive rate and false positive rate, respectively.
Figure 5: Qualitative results visualization of anomaly localization performance on MAD-Sim.
5.1 Pose-agnostic Anomaly Detection Benchmark on MAD Dataset
Our experiments provide a fair comparison between the state-of-the-art methods and OmniposeAD in the PAD setting, including pixel and image-level AUROC performance. Table 2 and Figure 5 clearly demonstrate that OmniposeAD achieves the state-of-the-art performance on MAD dataset for pose-agnostic anomaly detection evaluation. Specifically, OmniposeAD outperforms Feature Embedding-based CFlow[18] and Reconstruction-based UniAD[57] by a large margin,  $+7.0 / +19.6$  and  $+8.7 / +28.7$ , respectively. Its noteworthy that additional data (i.e., using a pre-trained model on ImageNet during the training phase) always brings advantage. This consistent superiority can be attributed to ImageNet's extensive repository of object representations spanning diverse poses. In a direct comparison with methods not utilizing additional training data, our approach showcases a remarkable enhancement, as evidenced by a substantial increment of  $+29.2 / +8.3$ . Both the Feature Embedding and Reconstruction-based methods, namely DRAEM[60] and Cutpaste[22], employ the generation of pseudo-anomalies as a data augmentation technique. However, these approaches
exhibit a notable decline in performance within the PAD setting. Objects may present different representations in different poses, leading to complex patterns of visual anomalies. It is difficult to effectively characterize the anomalies of objects under different viewpoints through simple data enhancement methods.
Our benchmarks demonstrate the considerable potential of both feature embedding-based and reconstruction-based approaches in the PAD setting. In particular, for the pixel-level anomaly localization task, the performance of STFPM[45](89.3), CFlow[18](90.8), CFA[21](89.8), FAVAE[51](89.4), and UniAD[57](89.1) shows a slightly fluctuating but largely respectable performance. However, with the exception of our proposed OmniposeAD, the performance of benchmark methods in image-level anomaly detection tasks is not satisfactory. It is worth noting that UniAD[57], although capable of learning multi-category anomaly boundaries using a single model, still struggles to achieve satisfactory image-level anomaly detection performance for multi-pose anomalies in the PAD setting. To address this gap, algorithms that are adept at capturing the normal attributes of multi-pose objects need to be explored to achieve reasonable image-level anomaly detection performance within the PAD setting.
5.2 Correlation of performance with object attributes
The results are shown in Figure.6, where the performance of most methods is positively correlated with color contrast and negatively correlated with structure complexity, which is consistent with our intuition. Notably, Cutpaste[22], a representative approach that generates anomalies and reconstructs them through a self-supervised task, stands out as being sensitive to color contrast but surprisingly tolerant towards shape complexity. Furthermore, we demonstrate the robustness of our proposed OmniposeAD to changes in object attributes.
Figure 6: Correlation between object attributes and anomaly detection performance.
Figure 7: Visualization of reference reconstruction between OmniposeAD and FAVAE.
5.3 Validation of reference reconstruction on OmniposeAD
Both the reconstruction-based method and our proposed OmniposeAD essentially reconstruct the normal reference. In the quantitative results of the anomaly localization task, the representative reconstruction-based method FAVAE achieved 89.4, and we further visualized the reconstructed references to explore the gap. Figure.7 shows that FAVAE does not reconstruct references for unseen poses well (the quantitative results do not align with visual facts), hinting that the existing evaluation metrics are potentially unsatisfactory in the PAD setting. Our further discussion is provided in Sec.4 of the Supplementary.
5.4 Effect of Dense-to-Sparse training data on OmniposeAD performance
In this section, we utilize the MAD-Real dataset to evaluate the impact of the transition from dense to sparse viewpoint training data on the performance of the OmniposeAD method, reflecting the lack of available viewpoints for real-world applications. We control the sample size of the training set for the OmniposeAD model to be 100, 70, and 50, respectively. As shown in Table 3: Despite the sparse training data leads to a slight decrease in performance, the less pronounced drop in pixel
Figure 8: Qualitative results visualization of OmniposeAD anomaly localization performance from multi-view on MAD-Real.
level performance suggests that OmniposeAD excels in capturing local consistency within objects. The more significant decrease at the image-level indicates that sparse data impacts OmniposeAD's ability to learn the holistic normal features of object from global perspective. We further showed the multi-view anomaly localization performance of OmniposeAD on the MAD-Real dataset in Figure.8.
Table 3: Dense-view to sparse-view object anomaly detection results on MAD-Real.
Sample Size Object Class	100	70	50	Sample Size Object Class	100	70	50
Image	Pixel	Image	Pixel	Image	Pixel	Image	Pixel	Image	Pixel	Image	Pixel
Gorilla	93.6	99.5	85.4	97.5	91.0	97.4	Pig	75.7	96.4	73.9	95.7	72.8	95.4
Unicorn	86.2	96.5	84.4	96.1	80.2	94.9	Zalika	88.2	99.1	78.8	96.8	72.8	96.0
Mallard	87.3	96.3	80.3	94.4	74.6	95.1	Phoenix	86.0	99.4	84.6	99.2	82.6	98.8
Turtle	99.1	95.4	95.1	92.3	83.4	90.4	Elephant	91.0	98.1	82.5	95.9	85.1	96.5
Whale	82.1	98.5	82.5	97.0	76.0	93.1	Parrot	91.5	99.1	85.5	98.2	87.9	96.9
Bird	92.0	95.1	91.6	91.7	89.1	93.4	Cat	78.9	97.6	73.5	97.6	73.7	97.1
Owl	89.1	99.3	78.8	98.1	89.1	98.8	Scorpion	87.4	94.2	77.0	91.8	74.3	91.5
Sabertooth	93.8	97.8	84.9	95.1	81.5	94.4	Obesobese	88.5	96.1	86.4	97.9	81.8	97.8
Swan	80.1	98.0	77.0	97.3	71.3	96.7	Bear	97.8	99.3	92.0	98.0	87.8	97.9
Sheep	84.3	97.6	83.2	97.6	82.3	92.8	Puppy	93.1	98.5	86.5	95.8	80.3	95.3
Conclusion. In this work, we introduce pose-agnostic anomaly detection setting, and develop the MAD dataset, which is the first exploration to evaluate pose-agnostic anomaly detection algorithms. In addition, we propose OmniposeAD to solve the problem of unsupervised anomaly detection in different poses of an object, which avoids missing potential occluded anomaly regions. Limitation. For the MAD dataset, which includes only standard rigid objects and no naturally varying objects, the complexity and diversity of industrial products made it difficult to collect and quantify a complete sample of diverse features. For OmniposeAD, it is difficult to generalize to novel classes to obtain appreciable performance; for all modules, we chose the vanilla method in pursuit of robustness, which has the potential for faster training times, less training data, and more realistic high-frequency details.
We would like to express our sincere appreciation to Shenzhen Qianzhi Technology Co.,Ltd for their support in providing Lego toys manufacturing and defect samples.