IGSPAD: Inverting 3D Gaussian Splatting for Pose-agnostic Anomaly Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 21:20:33

Abstract: Pose-agnostic anomaly detection refers to the situation where the pose of test samples is inconsistent with the training dataset, allowing anomalies to appear at any position in any pose. We propose a novel method IGSPAD to address this challenge. Specifically, we employ 3D Gaussian splatting to represent the normal information from the training dataset. To accurately determine the pose of the test sample, we introduce an approach termed Inverting 3D Gaussian Splatting (IGS) to address the challenge of 6D pose estimation for anomalous images. The pose derived from IGS is utilized to render a normal image well-aligned with the test sample. Subsequently, the image encoder of the Segment Anything Model is employed to identify discrepancies between the rendered image and the test sample, predicting the location of anomalies. Experimental results on the MAD dataset demonstrate that the proposed method significantly surpasses the existing state-of-the-art method in terms of precision (from 97.8% to 99.7% at pixel level and from 90.9% to 98.0% at image level) and efficiency.

Main paper: 
 introduction: 
 Due to the scarcity of anomalous samples, researchers have focused on unsupervised anomaly detection, where only normal samples are present in the training set. This approach has significantly advanced the field of visual anomaly detection. However, current unsupervised anomaly detection methods assume that the test images are well-aligned with the training images, a condition that only represents a subset of real-world scenarios. To further advance the field of visual anomaly detection, we should consider more common scenarios, such as Pose-agnostic Anomaly Detection (PAD). Following the work of Zhou et al. [27] , Pose-agnostic Anomaly Detection asserts that there is no precise alignment between training and test set images, with the test set potentially including poses that were never encountered in the training set, thus increasing the complexity of anomaly detection.The Multi-pose Anomaly Detection (MAD) [27] dataset is the first dataset specifically designed for Pose-agnostic Anomaly Detection (PAD). Testing on the MAD dataset, the efficacy of existing 2D unsupervised anomaly detection methods is significantly reduced. OmniposeAD [27] , the first method tailored for the PAD task, greatly improves detection performance compared to conventional 2D unsupervised methods. OmniposeAD utilizes NeRF [15] to represent the normal information of the training set. For each test image, OmniposeAD estimates the camera pose using iNeRF [21] and then renders a normal image based on the estimated pose. The test image and its corresponding rendered image are subsequently processed through feature extraction and compared to localize anomalies area. However, as reported by OmniposeAD, running the complete framework on the MAD dataset requires 10 to 15 hours on a single NVIDIA Tesla A100, indicating low time efficiency. Consequently, there is a pressing need to develop methodologies for the PAD task that not only improve performance but also enhance time efficiency.Recently, 3D Gaussian Splatting (3DGS) [10] has emerged as a popular method for 3D scene representation, achieving stateof-the-art results in rendering speed and image quality. With its significant advantages in efficiency and quality, 3DGS proves more effective for representing normal information compared to NeRF [15] . Additionally, the differentiability of 3D Gaussians allows for the calculation of gradients of the loss with respect to the camera pose, which enables the estimation of camera poses for test images, addressing the challenge of pose-agnostic in testing.To this end, we propose our method Inverting 3D Gaussian Splatting for Pose-agnostic Anomaly Detection (IGSPAD). Initially, IGSPAD trains 3DGS with normal images from the training set to represent the normal information. Unlike traditional Six-degree-offreedom pose estimation (6DoF) , estimating the camera pose for anomalous images would be disrupted by the anomalous regions. To address this issue, We specifically designed Inverting 3D Gaussian Splatting (IGS) to estimate the camera poses of the anomalous images. The test image and the normal image rendered under the estimated pose is then processed through a pre-trained feature extractor, and the cosine similarity of the features is calculated to generate the final anomaly detection results.Our proposed IGS involves three principal steps: Gradient Calculation: We explicitly calculate the gradients of the loss with respect to the camera pose using the chain rule.Initial Pose Estimation: Using LightGlue [13] , we match the test image against all images in the training set, adopting the camera pose information of the most closely matched training image as the initial pose.Matching Points Sampling: We sample around the points that match between the rendered and test images to minimize the influence of anomalous regions on the camera pose gradients.Subsequently, we evaluated and compared the efficacy of different feature extractors on anomaly detection performance, ultimately selecting the image encoder of SAM [11] as the feature extractor for our method.Extensive experiments on the MAD dataset demonstrated the effectiveness of our method. Our approach achieved state-of-theart performance in Pose-agnostic Anomaly Detection, enhancing the pixel-level AUROC from 97.8% to 99.7% and the image-level AUROC from 90.9% to 98.0%. Additionally, we achieved a pixel-level AP of 75.53%. Our method required only 5 hours on a 4090 GPU, marking at least a twofold improvement in time efficiency over OmniposeAD, which took 10 to 15 hours on an A100 GPU. The main contributions of our method are summarized as follows:• We introduce a novel method for Pose-agnostic Anomaly Detection called IGSPAD. [1] , MPDD [8] , BTAD [16] , and VisA [28] .2.1.2 Reconstruction-based Methods. Reconstruction-based methods, such as DRAEM [25] , FAVAE [20] , and UniAD [22] , typically operate under the assumption that models trained solely on normal samples struggle to effectively reconstruct anomalous regions. Consequently, areas in test images showing significant discrepancies before and after reconstruction are more likely to be identified as anomalous.2.1.3 Feature Embedding-based. Feature embedding-based methods, such as PaDiM [4] , PatchCore [17] , and Fastflow [23] , typically employ a pre-trained feature extractor to extract image features and then learn the representation of normal features. Consequently, regions in the test images that deviate significantly from the representation of the normal features are more likely to be identified as anomalous.2.1.4 Data Augmentation-based. Data Augmentation-based methods, such as Defect-GAN [26] , AnomalyDiffusion [7] and CAGEN [9] , address the challenge of insufficient anomalous samples by simulating or generating anomalies, thereby transforming the unsupervised anomaly detection task into a supervised segmentation or classification task. 
pose-agnostic anomaly detection: 
 As introduced by PAD [27] , pose-agnostic anomaly detection task operates under the assumption that the images in both training and test sets are not aligned well. The training set includes normal images of various poses along with their pose information, while the test set contains both normal and anomalous images without pose information. PAD introduces the OmniposeAD approach to address poseagnostic anomaly detection. Initially, OmniposeAD trained a NeRF [15] model using the training set. Given a test image, the camera pose is estimated using iNeRF [21] , and this estimated pose information is input into the trained NeRF model to render a normal image that aligns well with the test image. Then comparing these two images and localizing the anomalies.Recently, 3D Gaussian Splatting (3DGS) [10] has achieved tremendous success in 3D scene representation. As 3D Gaussians are differentiable, gradient descent can be employed to optimize the parameters of 3D Gaussians, achieving a compact 3D representation. Furthermore, the fast differentiable rasterizer for Gaussians enables 3DGS to render high-resolution images rapidly. Those capabilities allows 3DGS to demonstrate significant advantages in high-quality, real-time novel-view synthesis. 3DGS has been successfully applied in various domains, including 3D generation [3] and Simultaneous Localization and Mapping (SLAM) [14] .Segment Anything Model (SAM) [11] is a new paradigm for segmentation models, using point prompts and box prompts to indicate the content that the model needs to segment. Trained on a large segmentation dataset of over 1 billion masks, SAM is capable of segmenting any object on a certain image. Works such as Segment Any Anomaly (SAA) [2] are dedicated to applying SAM for anomaly detection. 
approach: 
 In this section, we introduce the proposed method IGSPAD. First, we state the definition of pose-agnostic anomaly detection. Subsequently, we detail our anomaly detection approach in two subsections. As shown in Fig. 2 , we employ 3DGS [10] to represent the normal information of the training set and then estimate the camera pose of the test image through inverting 3D Gaussian Splatting (IGS). To minimize the influence of anomalous regions on gradient backpropagation, we introduce the Matching Points Sampling (MPS) method. Finally, we extract features from both the rendered image and the test image, compute the cosine similarity for each pixel, and obtain the anomaly scores.The concept of pose-agnostic anomaly detection, as defined by PAD [27] , assumes that the test samples are not well-aligned with the images in the training set. Specifically, the training set comprises anomaly-free images of objects in the same class but captured in various poses, with attached pose information. In contrast, the poses of the test samples remain unknown and can be arbitrary, potentially including poses not previously encountered in the training set. This discrepancy makes it challenging to find a training image similar enough to a test sample for effective anomaly detection.To address this challenge, we first employ 3D Gaussian Splatting [10] to reconstruct the normal 2D images into a 3D representation, thereby capturing the 3D normal information. As a result, our objective transitions from locating a similar image in the training set to rendering an image from the 3D Gaussian model that aligns accurately with the test image. This task evolves into a 6D pose estimation problem. Drawing on the method similar to iNeRF [21] , Figure 4 : When backpropagating the camera pose, anomalous regions such as a missing tail can hinder optimization. By using matching point sampling, the impact of these anomalous areas can be eliminated.we optimize the camera parameters by computing the gradient with L1 loss respect to the camera pose. Our proposed method Inverting 3D Gaussian Splatting (IGS) comprises three principle steps.The gradient with respect to the camera pose. As the 3D Gaussian is differentiable, we are able to compute gradient of the loss with respect to the camera pose. Gaussian Splatting SLAM [14] provides the analytical Jacobian of the SE(3) camera pose with respect to the 3D Gaussians used in 3DGS. Instead of utilizing Lie algebra as in Gaussian Splatting SLAM, we directly compute the gradient of the translation component t of the view matrix T using the chain rule. In 3DGS, a pixel colour 𝐶 𝑝 can be calculated as follows:EQUATIONwhere 𝑐 𝑖 denotes the learned color of the 𝑖 𝑡ℎ Gaussian, and 𝛼 𝑖 is related to the 2D Gaussian 𝑁 (𝝁 ′ , 𝚺 ′ ) projected onto the camera coordinate system plane from the 𝑖 𝑡ℎ 3D Gaussian 𝑁 (𝝁, 𝚺) which can be calculated as follows:EQUATIONwhere P represents the projection operation, T denotes the view matrix, J is the Jacobian of the linear approximation of the projective transformation and W refers to the rotational component of the view matrix T. According to the chain rule, we have:EQUATIONAccording to Eq. ( 2),𝜕𝝁 ′ 𝜕t = J, which is easier to calculate. From this, we can explicitly calculate the gradient of the loss with respect to the translation component t of the view matrix T, and the calculation of the gradient for the rotational component follows the implementation used in Gaussian Splatting SLAM [14] .From the bad case shown in Fig. 8 , a poor initial pose can have a devastating impact on the performance of anomaly detection and it is important to find a good initial pose. We utilize the pose information from the training set, matching the test image with all images in the training set using LightGlue [13] . We assume that the more matched points two images have, the closer their poses are. The pose information from the training image with the highest number of matching points to the test image is used as the initial pose.Unlike traditional 6D pose estimation problem, pose estimation on anomaly images can be affected by anomalous regions. The losses caused by these anomalous areas can impact the optimization of the camera pose. As Fig. 3 demonstrate, IGS without MPS would focus on compensating for the losses caused by anomalies at an early stage. To address this issue, we propose Matching Points Sampling (MPS).We first matching the rendered image with the test image using LightGlue [13] . As shown in Fig. 4 , the anomalous area can hardly be matched to any point in the rendered image. Then we set the weights of the pixels within a 𝑘-pixel radius around the matching points to 1, while the rest are set to 0, resulting in the weighting map. The loss can be caculate as follows:EQUATIONwhere 𝐼 𝑟 and 𝐼 𝑡 refer to the rendered image and the test image, and 𝑊 is the weighting map. As shown in Fig. 4 , the sampled image consists mostly of normal regions, the anomalous regions excluded from backpropagation. Fig. 3 demonstrates that IGS with MPS consistently aligns with the normal regions.Given a test image 𝐼 𝑡 , we obtain a pose Θ through IGS and a normal rendered image 𝐼 𝑟 of Θ. Assuming that 𝐼 𝑟 is well-aligned with 𝐼 𝑡 , anomaly detection and localization then resemble a "spot the difference" task, where the objective is to identify discrepancies between 𝐼 𝑟 and 𝐼 𝑡 .Follow Feature-Embedding-based anomaly detection methods like PADIM [4] and PatchCore [17] , we obtain the feature map 𝑓 𝑡 ∈ 𝑅 𝐶 ×𝐻 ×𝑊 of 𝐼 𝑡 and the feature map 𝑓 𝑟 ∈ 𝑅 𝐶 ×𝐻 ×𝑊 of 𝐼 𝑟 . Then the pixel level anomaly score 𝑆 𝑃 and the image level score 𝑆 𝐼 between 𝐼 𝑡 and 𝐼 𝑟 can be calculate as follows:𝑆 𝑃 (𝑖, 𝑗) = 1 -f t (:, i, j) • f r (:, i, j) ∥f t (:, i, j)∥∥f r (:, i, j)∥(5)EQUATIONwhere 𝑖 𝑗 represents the position of pixels and 𝐴𝑃 () signifies the operation of average pooling. 𝑆 𝑃 will be upsampled to the size of the test image 𝐼 𝑡 . We consider different pretrained models as our feature extractor, ResNet18 [6] , WideResNet50 [24] , CAIT [18] , and the image encoder of SAM [11] . ResNet18, WideResNet50 and CAIT are pre-trained on classification tasks, whereas SAM is pre-trained on segmentation tasks, which is more closely related to the anomaly detection task. Table 1 demonstrates that, compared to other pre-trained models, SAM significantly enhances the capability of anomaly detection, especially the pixel AP metric. Consequently, we ultimately select SAM as our feature extractor. 
experiments: 
 In this section, we first introduce our experimental setup and implementation details. Then, we conduct ablation studies on the individual components of IGSPAD. Finally, we compare our method with other existing Pose-agnostic Anomaly Detection methods.4.1.1 Dataset. MAD [27] , which means multi-pose anomaly detection dataset, provides 20 categories of LEGO animals toys with diverse shape complexity and color contrasts, constructed using Blender in combination with Ldrew (LEGO parts library). For each category, the training set includes 210 images of different poses along with their corresponding pose information, while the test set comprises 150 to 300 anomalous images and several normal images without pose information. All anomalies are manually generated by professional LEGO players using Photoshop, resulting in sufficiently realistic anomalies. Furthermore, MAD quantifies color contrast and shape complexity for each category in order to assess their influence on the performance of anomaly detection. Detailed information is provided in the appendix. It is important to note that none of the data are derived from the real-world scenarios.Following previous work, we select the Area Under the Receiver Operating Characteristic Curve (AUROC) as our primary evaluation metric. Additionally, we specifically report the pixel-wise average precision (AP) to demonstrate the accuracy of our method in anomaly segmentation. In this subsection, we discuss the effectiveness of MPS and the differences between various feature extractors.As Table 1 illustrates, the MPS enhances the performance of anomaly detection in all metrics. The augmentation in AP is notably significant. Since most of rendered image without MPS already achieve satisfactory alignment with the test images, the improvement in the AUROC metric is relatively modest. Fig. 5 showcases instances of suboptimal alignment under the regime without MPS, highlighting that such misalignment can precipitate a significant number of false positives in anomaly localization, thereby exerting a considerable detrimental impact on the AP metric.Fig. 3 demonstrates the alignment of rendered images and test images in the same iteration step under the conditions of IGS without MPS and IGS with MPS. In the early stages of iteration, IGS without MPS attempts to align the anomalous regions with the normal regions. By the 250th iteration step, IGS with MPS has already aligned, while IGS without MPS still has not aligned by the 300th step.Compared to simple pose estimation, our method MPS can avoid the influence of anomalous regions on the gradient of camera pose, accelerate the optimization process of camera pose estimation, achieve more accurate camera pose estimation, and significantly improve the performance of anomaly detection. 1 , different feature extractors have a significant impact on anomaly detection performance, especially in the AP metric. Not surprisingly, sam, which is pre-trained on segmentation tasks, has a substantial advantage over resnet18, wideresnet50, and CAIT, which are pre-trained on classification tasks. Fig. 6 presents the visualization results of anomaly localization with different feature extractors. Sam has demonstrated excellent results in anomaly segmentation, showing heightened sensitivity to anomalies as well as robustness to noise in non-anomalous regions. Comparing three different models of SAM with varying amounts of parameters, the model with the fewest parameters surprisingly achieved the best anomaly detection performance. This indicates that it is possible to enhance the time efficiency of anomaly detection without reducing its effectiveness. Fig. 7 compares the performance of different feature extractors at varying levels of color contrast and shape complexity. Contrary to most methods described in PAD [27] , our method's performance is negatively correlated with color contrast and positively correlated with structural complexity. Comparing the performance of different feature extractors, there is an overall negative correlation with color contrast for all four types, with ResNet, WideResNet, and CaiT showing more stable performances, which might be more closely associated with IGS. In terms of shape complexity, ResNet, WideResNet, and CaiT demonstrate a negative correlation, while SAM shows a positive correlation, underscoring the advantage of SAM in dealing with complex scenarios. With the increase in shape complexity, the other three feature extractors exhibit a rise in false positives unrelated to anomalies, leading to a reduction in anomaly detection performance.Overall, SAM as the feature extractor, consistently outperforms the remaining three across all categories, which provides a directive for future training of specialized pre-trained feature extractors dedicated to anomaly detection.For a comprehensive comparison, We selected advanced 2D image anomaly detection methods and the SOTA pose-agnostic anomaly detection method OmniposeAD [27] . For the Feature Embeddingbased 2D anomaly detection methods, we considered PatchCore [17] , STFPM [19] , Fastflow [23] , CFlow [5] and CFA [12] . For the Reconstruction-based 2D anomaly detection methods, we selected DRAEM [25] , FAVAE [20] , and UniAD [22] .Table 2 demonstrates the superiority of our method, which achieves the best performance in almost all categories, except for the image-level AUROC in the Obesobeso. Compared to the current state-of-the-art method, we have achieved an improvement of 1.9 in average pixel-level AUROC and 7.1 in average image-level AUROC. Compared to the method OmniposeAD which also based on rendering and pose estimation, our approach achieves more accurate camera pose estimation and renders higher quality images. Additionally, by applying SAM as a feature extractor, our method significantly improves segmentation accuracy, thereby enhancing its exceptional performance in anomaly detection.Fig. 8 displays a series of qualitative examples. The results of anomaly prediction are satisfactory most of the time. However, there are still some bad cases where the edges match the test image, but the actual rendering is of its backside or from another angle. These issues are typically due to optimization commencing from an unfavorable initial pose, leading the loss to settle into a local minimum, indicating that a good initial camera pose is extremely important.  
conclusion: 
 In this paper, we propose a novel method IGSPAD for Pose-agnostic Anomaly Detection, which is based on the 3D Gaussian Splatting. We utilize 3DGS to represent the normal information of training set.For the problem of camera pose estimation in anomalous images, we propose the method of Inverting 3D Gaussian Splatting with Matching Points Sampling, which eliminates the impact of anomalous regions on camera pose estimation. We compared various feature extractors and analyzed the impact of features on the performance of anomaly detection. Extensive experiments on the MAD dataset show that, compared to existing state-of-the-art methods, our approach achieved significant improvements. In future work, we plan to further apply our method to anomaly detection in real-world scenarios, to validate whether a training set generated using our approach can effectively detect anomalies in real scenes. Additionally, we will explore how to train a pre-trained feature extractor specifically designed for anomaly detection.  
 
 Appendix:  


Decision: Despite on borderline reject from one of the reviewers, the AC concurs with the two positive reviewers and recommend for publication of this paper. However, the authors should consider to release their code to the public as it is an industry standard nowadays. The authors should address all issues raised by the reviewers in their final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper focus on the task of pose-agnostic anomaly detection, where the pose of test samples is inconsistent with the training dataset, allowing anomalies to appear at any position in any pose. To tackle this research problem, this paper proposes an Inverting 3D Gaussian Splatting (IGS) and pose derived from IGS is utilized to render a normal image well-aligned with the test sample. Then image encoder of the Segment Anything Model is employed to identify discrepancies between the rendered image and the test sample, predicting the location of anomalies.
strength_weakness: strength_weakness: 1. The primary and most notable strength of this paper is its focus on a novel dataset, Multi-pose Anomaly Detection (MAD), which addresses a new research problem. The MAD dataset is the first specifically designed for Pose-agnostic Anomaly Detection (PAD). Consequently, this work represents a pioneering effort in this area.

2. This paper heavily relies on the recent method of 3D Gaussian Splatting, which is popular and effective for 3D scene representation. It introduces a novel variant, Inverting 3D Gaussian Splatting, which effectively generates the final anomaly detection results.

3. The experiments conducted on the newly released dataset are comprehensive, and the results are outstanding.
limitations: limitations: For a very new task, without established baseline methods (this work would be the first one), it is challenging to identify the limitations of this paper by comparing it with previous methods. Despite this, I find the article's approach simple, logical and effective, with good visual presentation and writing quality, leading me to lean towards acceptance.
 
There is an error in the citation: PAD is published in NeurIPS 2023, not 2024 (line 1004).
paper_summary: paper_summary: This paper proposes a novel method, IGSPAD, for pose-agnostic anomaly detection. It uses 3D Gaussian Splatting to represent the normal information of the training set. To address the problem of camera pose estimation in anomalous images, the paper introduces Inverting 3D Gaussian Splatting with Matching Points Sampling, which eliminates the impact of anomalous regions on camera pose estimation. Extensive experiments on the MAD dataset show that the proposed approach achieves new state-of-the-art results.
strength_weakness: strength_weakness: (1)	The paper is, for the most part, well-written and easy to follow.
(2)	The method is novel and effective.
(3)	There is strong experimental evaluation as well as comparisons to baselines.
limitations: limitations: In Table 1, it would be beneficial to include experimental results using the same feature extractor as in reference [27].
paper_summary: paper_summary: This paper proposes a novel method called IGSPAD (Inverting 3D Gaussian Splatting for Pose-agnostic Anomaly Detection) for pose-agnostic anomaly detection. The key contributions include: 1) Using 3D Gaussian Splatting (3DGS) to represent normal information from the training dataset. 2)Introducing Inverting 3D Gaussian Splatting (IGS) with Matching Points Sampling (MPS) to accurately estimate the pose of test images while avoiding the influence of anomalous regions. 3) Leveraging the image encoder of the Segment Anything Model (SAM) as a feature extractor to identify discrepancies between rendered normal images and test images. 4)Achieving state-of-the-art performance on the MAD dataset, significantly improving precision and efficiency compared to existing methods.
strength_weakness: strength_weakness: 1. The proposed IGS with MPS approach provides an innovative solution to the key issue of estimating camera pose in the presence of anomalies.

2. The method achieves impressive state-of-the-art results on the MAD dataset, showing substantial improvements in both precision and efficiency over prior work.
limitations: limitations: 1. The writing quality of this paper needs considerable improvement. The authors fail to adequately summarize their main contributions in the introduction and conclusion sections. The abstract does not provide a strong motivation for the proposed work. It appears that the related work section is merely a list of existing studies without a critical analysis.

2. Lack of comprehensive comparisons and discussions with other anomaly detection methods in the field, such as reconstruction-based, feature embedding-based, and data augmentation-based methods. Although comparisons with 2D methods and OmniPoseAD are provided, the paper lacks experimental comparisons with more state-of-the-art methods, especially some recent transformer-based approaches, on both the MAD dataset and other commonly used anomaly detection benchmarks. This makes the experimental section appear less comprehensive.

3. Lack of systematic ablation studies evaluating the effects of different MPS settings. MPS is a key step, but the paper lacks systematic ablation experiments on hyperparameter settings such as the sampling radius and the number of sampling points in MPS. Different sampling settings may significantly influence the robustness to various anomaly patterns (area, distribution, type).