One-to-Normal: Anomaly Personalization for Few-shot Anomaly Detection

Authors: yiyue  li
shaoting  zhang
kang  li
qicheng  lao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:50:51

Abstract: Traditional Anomaly Detection (AD) methods have predominantly relied on unsupervised learning from extensive normal data. Recent AD methods have evolved with the advent of large pre-trained vision-language models, enhancing few-shot anomaly detection capabilities. However, these latest AD methods still exhibit limitations in accuracy improvement. One contributing factor is their direct comparison of a query image's features with those of few-shot normal images. This direct comparison often leads to a loss of precision and complicates the extension of these techniques to more complex domains—an area that remains underexplored in a more refined and comprehensive manner. To address these limitations, we introduce the anomaly personalization method, which performs a personalized one-to-normal transformation of query images using an anomaly-free customized generation model, ensuring close alignment with the normal manifold. Moreover, to further enhance the stability and robustness of prediction results, we propose a triplet contrastive anomaly inference strategy, which incorporates a comprehensive comparison between the query and generated anomaly-free data pool and prompt information. Extensive evaluations across eleven datasets in three domains demonstrate our model's effectiveness compared to the latest AD methods. Additionally, our method has been proven to transfer flexibly to other AD methods, with the generated image data effectively improving the performance of other AD methods.

Main paper: 
 introduction: 
 Anomaly Detection (AD) has garnered considerable attention due to its wide applicability across various domains, such as industrial defect detection [20, 30, 2, 14, 27] , medical diagnostics [37, 21, 43] , video surveillance [22, 33] , manufacturing inspection [39, 9] . This heightened interest is primarily attributed to its only reliance on normal samples and its adoption of an unsupervised learning paradigm, where it typically learns from the distribution of normal samples to identify anomalies by detecting outliers [46, 39] . Traditional AD methods, including auto-encoder based [43, 40] , GAN-based, and knowledge-based approaches [3, 8, 35] , and others. Moreover, some diffusionbased methods [24, 11] have also emerged recently. Although most of these methods do not require annotated data, they do necessitate a substantial number of normal samples during the training phase to capture the distribution of normal samples effectively. However, this requirement has also severely constrained its advancement in various fields.Recent studies [36, 14, 38] in few-shot scenarios have improved AD tasks. Currently, state-ofthe-art (SOTA) advancements [45, 16, 10, 44] are primarily due to the development of large pretrained Visual-Language Models (VLMs). For example, WinCLIP [16] first uses pre-trained CLIP, employing carefully designed text prompts and image feature comparisons to perform few-shot anomaly detection. AnomalyGPT [10] eliminates the need for manually setting thresholds and supports multi-round dialogues. InCTRL [45] achieves general anomaly detection through incontext residual learning. Despite these innovations, the latest SOTA methods often rely on direct feature matching between few-shot normal images (i.e., reference images) and the query sample. However, without deeply exploring their subtle features, it is often difficult to achieve precise feature comparisons, which can easily lead to unstable results, since the non-anomalous differences between query and references can severely impact the prediction accuracy. Furthermore, the limited number of normal images serving as independent references also makes their prediction insufficient.In this work, we hypothesize that: 1) To achieve more accurate prediction results, it is essential to compare the query image with its corresponding or most similar normal image. Ideally, this comparison should involve a one-to-one transformation of the query image into its normal counterpart;2) Additionally, one should also aim for robust and stable results, necessitating a comprehensive approach to accurately predicting results from multiple perspectives. To this end, we propose an anomaly personalization method for few-shot anomaly detection. To achieve more accurate, personalized comparative predictions, as mentioned in 1), our method employs a diffusion model to create an anomaly-free customized model inspired by [31] . This model uses pairs of object text prompts and few-shot normal images (i.e., reference images) to explore the distribution of normal samples. Previous works [23, 4, 41] have demonstrated the effectiveness of such customized models. Furthermore, the suitability of diffusion models stems from their powerful customization capabilities, allowing for flexible control over intermediate steps and the generation process compared to other methods [26] . Additionally, for enhanced stability and robustness as mentioned in 2), we propose the use of triplet contrastive anomaly inference, i.e., in addition to comparing the query image with normal samples, we conduct comprehensive comparisons with personalized samples and text prompts. Furthermore, we use comprehensive state words and templates as text prompts inspired by [15, 16] . Ultimately, we synthesize predictions from diverse perspectives to yield the final anomaly score.To summarize, this paper makes the following main contributions:• We introduce a novel anomaly personalization method for few-shot anomaly detection, unlike other state-of-the-art (SOTA) approaches that directly compare query images with reference images, our method enables a finer-grained comparison through one-to-normal personalization of query images, leading to enhanced prediction accuracy.• To achieve more stable and robust results, we propose a triplet contrastive anomaly inference strategy. This approach facilitates a more comprehensive comparison by incorporating not only customized comparisons but also comparisons with anomaly-free samples and text prompts. The anomaly-free samples augment normal images sampled by an anomaly-free customized model.• Our proposed method demonstrates strong generalizability. We conduct comprehensive experiments across 11 datasets spanning three distinct domains: industrial, medical, and semantic. Moreover, the anomaly-free samples generated by our method can be used to augment the normal samples in most few-shot anomaly detection methods, enhancing the performance of some existing methods and demonstrating adaptability and flexibility.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Yes, the main claims presented in our abstract and introduction accurately reflect the contribution and scope of our paper Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
anomaly detection: 
 Anomaly detection (AD) is a critical task in computer vision aimed at identifying samples that deviate significantly from the norm. Traditional AD methods can be categorized into several types: auto-encoder based [43, 40] , GAN-based, and knowledge-based approaches [3, 8, 35] , and others. Recent advancements in AD research focus on few-shot and zero-shot learning to overcome data limitations. Few-shot AD methods like RegAD [14] utilize pre-trained models and a few normal samples from the target domain to detect anomalies without extensive re-training. WinCLIP [16] meticulously designs a variety of prompts to ensure the model comprehensively covers all possible normal and abnormal scenarios. AnomalyCLIP [44] uses CLIP for zero-shot anomaly detection. They have also meticulously designed prompts but have overlooked the aspect of image comparison. In summary, many of the current few-shot AD approaches have primarily focused on text prompt refinement without detailed exploration in visual feature comparisons and sample generation.Personalized text-to-image generation has emerged as a pivotal area, particularly for creating highly personalized and contextually accurate images. DreamBooth [31] is the first to focus on fine-tuning diffusion models with specific subject images to generate high-fidelity renditions of those subjects, typically using 3 to 5 images for customization and incorporating prior preservation to prevent language draft. SuTi [4] leverages a single model to learn from a multitude of expert models, each fine-tuned to specific subjects, allowing for instant personalization using in-context learning with only a few examples. Another innovative approach [23] involves "concept neurons" in diffusion models, identifying clusters of neurons that correspond to specific subjects within a pre-trained model. These methods have achieved significant results in image generation. However, most of these methods have been applied to image editing and synthesis, and have yet to be explored for anomaly detection.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification:Yes, our paper discusses the limitations of the work in the discussion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 We propose a novel anomaly personalization method to improve few-shot anomaly detection. First, to obtain more precise results, we aim for a one-to-one customized comparison, where the query image is compared with the image derived from itself, transformed towards an anomaly-free distribution to align the anomalous distribution with the normal state. This is achieved by employing an anomaly-free customized model (Sec. 3.2) followed by one-to-normal personalization of the query image (Sec. 3.3). Second, to enhance the detection stability and robustness, we further propose a triplet contrastive anomaly inference process (Sec. 3.4) , where the query image is thoroughly compared with its personalized version, augmented anomaly-free samples, and text prompts for comprehensive analysis.To develop a customized model for normal objects, we choose diffusion model [12] as the framework, building on the work based on [31] which has proven effective for specific object customization. To enhance the diversity of normal object samples given limited data, we first apply data augmentation to normal reference images. Subsequently, to better align objects with their textual descriptions, we provide the diffusion model with a series of demonstrations C o , using augmented reference images paired with their corresponding texts for specific objects. Specifically, for each object (e.g, cable), we prepare a series of augmented reference images (i.e., few-shot normal samples) x normal ∈ D train and their matching text c, i.e., a photo of normal [object], for tailored training of the diffusion model.Here, the specific category name (e.g., 'cable') is used to replace 'object'. Additionally, another set of demonstrations C o is employed to design a class-specific prior preservation loss, aimed at promoting diversity and preventing language drift, similar to [31, 4] . To obtain an anomaly-free customized model D θ (x normal,t , c) parameterized by θ on an object o, we constrain a pre-trained diffusion model on the image cluster C o with the denoising loss as:EQUATIONwhere c is a short description of normal images x normal , and x normal,t is a latent version of the input noised by t steps. Note that the dimensionality of both the image and the latent is the same throughout the entire process. Therefore, the anomaly-free customized model learns the distribution of normal images P normal conditioned on the text prompt c, where anomaly-free samples can be generated.Our proposed one-to-normal strategy is designed to transform the query image of an object toward the distribution of its normal samples. It adaptively retains the normal characteristics while the anomaly parts are gradually transformed with no defects. This is similar to previous work [23, 26] for image editing; however, our strategy specifically focuses on utilizing it to align with the normal manifold.Specifically, we first design text prompts to maximize the retention of information in the normal regions of the query image while transforming anomalous areas towards a normal state. To mitigate the influence of other factors (e.g., contrast, image quality), we simulate all potential normal states using these prompts. Inspired by [16, 44] , we curate a template list for various potential image physical states, e.g., a low-resolution photo of a normal object. Additionally, we include descriptors for common states shared by most normal objects, such as a photo of normal object without flaw. We provide these descriptors for normal state prompts in Appendix A.1.Given a series of normal state prompts {c} n i=1 described above, we aim to select the generated image that most closely resembles the normal state for the one-to-normal personalization of the query image x q . Specifically, we reconstruct a set of reconstructed images {x i } n i=1 where each text prompt c i guides the diffusion process of the customized anomaly-free model. We select the optimal text prompt c q that most closely resembles the normal state from the prompt set:EQUATIONwhere D θ represents the anomaly-free customized diffusion model derived from the Section 3.2, and L represents SSIM. Finally, we obtain the personalized images xq :EQUATIONwhere α is a noise schedule, ϵ N (0, I) represents Gaussian noise, and t is the diffusion step.Following [25] , during this noising phase, the hyperparameter t is meticulously adjusted to dictate the extent of the one-to-normal personalization, which determines the similarity between query image x q and personalized image xq . A relatively lower t means insufficient personalization where a substantial portion of the original query is retained. As t approaches 0, the generated image becomes identical to the input query image, and the text prompt's condition has no effect. Conversely, setting t to T initiates a complete forward diffusion process, consequently erasing all information in x q and allowing the generation to be fully guided by text prompts c q toward the normal manifold. Ultimately, we bridge the distributions of P query and P normal for obtaining P personalized , from which we can sample personalized images for subsequent anomaly detection tasks.To achieve comprehensive predictions and enhance the precision and robustness, we introduce triplet contrastive anomaly inference, i.e., in addition to comparing the query image with anomaly-free samples and text prompts, we also compare it with our personalized images. Finally, we integrate predictions from three comparison aspects to mutually complement each other.One-to-one personalized comparison. We divide the CLIP image encoder into n multi-feature extraction blocks, each designed to capture the intermediate features of the input image at different levels. For a query image x q and its corresponding personalized image xq , the model extracts features F q ∈ R h×w×d from x q and Fq ∈ R h×w×d from xq . The comparison score S P between the query image and its personalized image is then computed by evaluating their similarity in the extracted features at each level:EQUATIONwhere l denotes the l-th level feature extraction block, n represents the total number of blocks, and ⟨•⟩ signifies the cosine similarity function, G represents the grid number.Anomaly-free sample comparison. Our anomaly-free sample pool comprises a set of normal reference images and generated normal images, better representing the distribution of normal samples. We first generate normal samples from the customized anomaly-free model D θ in Section 3.2 and incorporate them into the anomaly-free sample pool for subsequent prediction tasks, as previous studies have demonstrated the diffusion model's capability to synthesize high-fidelity images, which have been proven effective in many works [41, 1, 25] . To further expand the anomaly-free sample pool, we use the same process described in Section 3.3 but with settings that more closely align with the distribution of normal reference image instead of the query image. Then, we employ the same CLIP image encoder to extract multi-level features from anomaly-free samples (e.g., 30) and store these features in a memory bank M . The prediction score S N between the query image and anomaly-free samples can be expressed as:EQUATIONText prompt comparison. To calculate the anomaly score between the query image features and the text prompt features, we categorize text prompts into two types: normal and abnormal objects, similar to [16, 5, 15] . We aim to cover more possible states for these objects to better simulate the various potential conditions of the images, similar to [16] . Specifically, the text features F text ∈ R 2×d are obtained using the CLIP text encoder for a set of given normal and abnormal prompts. To facilitate comparison, a linear layer is used to map the multi-level image features from the query image into F ′ q,l . These two features from text prompts and the query image are then used to calculate the anomaly score as follows:EQUATIONHere, sof tmax(•) is the average softmax scores across multiple levels.The final prediction result is generated by combining outputs from three branches: the personalized image, anomaly-free samples, and text prompts. These three sets of output scores provide complementary information for collaboration. The final anomaly score can be obtained by combining the three branches:EQUATIONwhere α, β are the hyper-parameters.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: Yes, the data in our paper is publicly available, and we will also make the code public in the future. Guidelines: Justification: Yes, we discuss both potential positive societal impacts and negative societal impacts of the work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment setup: 
 Dataset and Evaluation Metrics. We validate the effectiveness of our method across 11 datasets spanning three distinct domains: industrial, medical, and semantic domains. In the industrial domain, we utilize multiple datasets including MVTec-AD [2] , Visa [46] , KSDD [34] , AFID [32] , and ELPV [7] . For the medical domain, we incorporate datasets covering various modalities such as magnetic resonance imaging (MRI), computed tomography (CT), and optical coherence tomography (OCT). Specifically, the medical datasets include OCT2017 [17] , BrainMRI, HeadCT [13] , and RESC [13] . For semantic anomaly detection, we employ two datasets: MNIST [19] and CIFAR-10 [18] . These semantic datasets are utilized following the one-vs-all strategy, where one class is designated as normal, and all other classes are treated as anomalous. We follow [45] for the dataset partitions. We only use data from the MVTec-AD dataset as auxiliary training data. When testing on the MVTec-AD dataset, we use auxiliary data from the Visa dataset. The anomaly detection performance is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC) and the Area Under the Precision-Recall Curve (AUPRC).Competing Methods and Baselines. We compare our proposed method with various state-of-the-art anomaly detection (AD) methods, including traditional AD approaches (full-shot) such as PaDiM [6] and PatchCore [30] , both of which are adapted to the few-normal-shot setting. Additionally, we compare a traditional few-shot learning method, RegAD [14] and prompt learning method CoOp [42] , along with the latest AD methods based on vision-language models, including WinCLIP [16] and InCTRL [45] . To further validate the flexibility and applicability of our method, we evaluate our generated anomaly-free samples using four state-of-the-art anomaly detection methods: PaDiM, PatchCore, WinCLIP and InCTRL.Implementation Details. We select Stable Diffusion V1.5 [29] and utilize the CLIP with ViT-L/14 architecture [28] for all tasks. Our anomaly-free customized model is fine-tuned using Dreambooth [31] , and all model parameters are frozen in subsequent tasks. All the experiments are trained by use of PyTorch on an NVIDIA GeForce RTX 4090 GPU. Our anomaly detection task is a few-normal-shot setting including 2-shot, 4-shot, and 8-shot scenarios using only normal images. The ratio of the t-step is set to 0.3. We set the parameters α and β for A score to 1 and 0.5, respectively, across all datasets. The image-level memory bank is set to 30, and the inference resolution is 240×240. During inference, we use the same text prompts as WinCLIP [16] . To validate the improvement of other anomaly detection methods with our generated data, we use our generated anomaly-free samples to augment the datasets for these methods. For the MVTec dataset, we generate 100 anomaly-free samples per subclass to expand the reference dataset, while keeping all other settings identical to previous works.Comparative Analysis with Industrial Datasets. In Table 1 , we comprehensively compare five different datasets from the industrial domain, with the MVTec-AD and Visa datasets containing 15 and 12 subclasses, respectively. We also conduct an in-depth analysis across various few-shot scenarios (i.e., 2-shot, 4-shot, 8-shot) to verify the robustness of our method. The experimental results demonstrate that our method outperforms other approaches, showing superior performance, especially in the 4-shot and 8-shot scenarios, where it achieves the best results across all datasets. In the 8-shot setting, our method achieves higher AUCs by 3.4%, 4.1%, 0.6%, 1.2%, and 0.9% on the KSDD, ELPV, AFID, KSDD, Visa, and MVTec datasets, respectively, compared to the second-best baseline. Notably, when compared with the latest InCTRL method on the AFID dataset, our method achieves higher AUCs by 2.2%, 3.6%, and 4.1% in the 2-shot, 4-shot, and 8-shot settings, respectively. Additionally, the results indicate that our method is less affected by different runs, with variance significantly lower than other methods, supporting its superior stability and robustness. Finally, as shown in Figure 2 , the anomaly map indicates that our method accurately identifies abnormal regions Table 1 : A quantitative comparison of our proposed method and other methods using (AUROC (%), AUPRC (%)) as the evaluation metric on industrial datasets. We show the average performance and standard deviation across five runs, with the top value highlighted in bold for each comparison.PaDiM [6] Patchcore [30] RegAD [14] CoOp [42] WinCLIP [16] InCTRL [45] Ours MVTec (78.5±2. while being less likely to falsely recognize normal regions as anomalous. This further validates the accuracy of our approach. More anomaly maps and localization results are provided in Appendix A.2.Comparative Analysis with Medical Datasets. In Table 2 , we compare four medical image datasets from three different modalities (CT, MRI, OCT). Our method achieves superior few-shot anomaly detection performance across these diverse modalities, outperforming other methods in most datasets, particularly on the OCT2017 and RESC datasets. Notably, on the RESC dataset, our method surpasses the second-best method by 3.9%, 5.9%, and 4.6% in the 2-shot, 4-shot, and 8-shot settings, respectively. The suboptimal performance of other baseline methods on these datasets may be due to the unique characteristics of the medical domain, where much of the knowledge is not well-explored in VLMs pre-trained mostly on non-medical data. These results demonstrate that our proposed anomaly personalization strategy alleviates these domain-specific challenges more effectively. Additionally, because our method is highly effective in specific domains, it demonstrates greater stability compared to other methods, further proving its robustness.Comparative Analysis with Semantic Datasets. Here, we compare two semantic datasets using the one-vs-rest strategy, where one class is considered normal, and all other classes are treated as anomalous. The results show that our method achieves the best performance across three different few-shot settings. In the 8-shot scenario, our method reaches AUROC scores of 93.6(%) and 94.9(%) on the MNIST and CIFAR-10 datasets, respectively. These experiments demonstrate that our method is also more effective on semantic datasets compared to other baselines. Table 3 : A quantitative comparison of our proposed method and other methods using (AUROC (%), AUPRC (%)) as the evaluation metric on semantic datasets. We show the average performance and standard deviation across five runs, with the top value highlighted in bold for each comparison.Patchcore [30] RegAD [14] CoOp [42] WinCLIP [16] InCTRL [45] Ours Visualizations of Anomaly Personalization. Figure 3 compares the normal images (i.e., reference images), query images, and our resulting personalized images. As depicted, there are some differences between the normal image and the query image in terms of non-anomalous features, such as position, shape, and texture. Our personalized image retains most of the normal regions from the query image, with the anomalous regions largely converted to normal regions. This visualization further substantiates the precision of our method compared to existing few-shot AD approaches.The effectiveness of generated anomaly-free samples. To validate the effectiveness of our method in generating normal samples, we apply it to multiple AD methods, including Patchcore, RegAD, WinCLIP and InCTRL. As shown in the first row of Figure 4 , our method demonstrates improvements in the 2-shot, 4-shot, and 8-shot settings across all three methods. Notably, for the RegAD method, the AUC is improved by 6.8(%), 8.7(%), and 10.2(%), respectively. Additionally, we compare our generated strategy with five other AD datasets. After incorporating our anomaly-free samples, the four AD methods show varying degrees of improvement on industrial and medical datasets (Visa, KSDD, AFID, OCT2017, and headCT). In particular, the Patchcore and RegAD methods exhibit significant improvements on the OCT2017 dataset, with increases of 10.9(%) and 8.6(%), respectively. While InCTRL has already achieved high performance, our method still provides a certain level of enhancement. This experiment demonstrates that the data generated by our method can enhance existing AD methods, proving the flexibility and adaptability of our approach.The effectiveness of the triplet contrastive anomaly inference. Here, we discuss the impact of each branch in our triplet contrastive anomaly inference strategy, analyzing datasets across three different domains. Table 4 validates the effectiveness of our strategy, showing that it is optimal in most datasets.When only text prompts are used (S text ), meaning there is no reference image input, particularly low results are observed in the medical datasets OCT2017 and RESC. However, when reference images are supplied (either S P or S N ), a significant improvement is observed. This might be because the model lacks an inherent understanding of specific medical domains, thus requiring normal images as references. When each strategy is used individually, we find that most results are not as good as when the strategies are combined, indicating that these strategies complement each other. When all three strategies are employed together, most datasets demonstrate optimal performance. However, in certain cases, such as the KSDD dataset-which focuses on surface defect inspection-utilizing all scores does not consistently yield the best results. This inconsistency may stem from the high diversity in the distribution of normal images, which presents challenges for the AD model's learning process. Nevertheless, across the majority of datasets, the combined use of three strategies leads to relatively strong performance, showcasing that our triplet contrastive anomaly inference strategy effectively enhances accuracy.The effectiveness of text prompts in anomaly personalization. We also conduct ablation experiments on the settings of the text prompts c q used in the one-to-normal personalization stage (Section 3.3). In this work, we employed three prompts to generate three images (one for each prompt). The results in Table 5 demonstrate that our designed text prompt strategy is effective for one-to-normal personalization, where most datasets show improvement.Inference time. Regarding inference time, our proposed method is slightly higher (+200-300ms per query image) than that of WinCLIP (389ms) and InCTRL (276ms). If necessary, we can further increase the inference speed by reducing the number of generated samples or decreasing the memory bank size. When using a single prompt corresponds to generating only one personalized image, the required inference time (326ms) is slightly lower than that of WinCLIP, while still demonstrating superior performance compared to other methods. 
conclusion: 
 We introduce a novel personalized few-shot anomaly detection method that enhances prediction accuracy through one-to-normal personalization of query images. Unlike other state-of-the-art approaches that directly compare query images with reference images, our method enables a finergrained comparison. Our triplet contrastive anomaly inference strategy further stabilizes results by incorporating personalized images, anomaly-free samples and text prompts, facilitating a more comprehensive comparison. Extensive experiments across 11 datasets in industrial, medical, and semantic domains demonstrate the method's generalizability and effectiveness. Moreover, the anomaly-free samples generated by our method can augment the normal samples in existing few-shot anomaly detection techniques. Experimental results also demonstrate that our method improves the performance of several existing few-shot anomaly detection techniques.Limitation Achieving precise detection necessitates a comprehensive exploration of the distribution of each category, which in turn requires a few reference images from each category. Consequently, our current method is not applicable to zero-shot scenarios. Future research will focus on enhancing the capability to identify anomalies in zero-shot settings more accurately, as well as exploring real-time applications and open-vocabulary scenarios.Broader Impacts This article focuses on a customized few-shot anomaly detection method, which offers more precise and stable anomaly detection. It has the potential to enhance development in industrial and medical fields. There are no negative societal impacts involved in this work. NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our experiments do not involve this aspect.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, the assets used in the paper (e.g., code, data, models) are all properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: All reviewers are positive about this paper. They acknowledge the novelty of this paper and the achieved superior performance. They also have some questions about the ablation study and other analysis issues, which are well-addressed during the rebuttal. Therefore, the AC agrees with the positive opinion of the reviewers and recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper focuses on a practical yet challenging anomaly detection in a few-shot-normal-image setting. Instead of directly matching features between the query image and a few normal reference images, the core insight is to replace the reference image with a personalized normal image generated by an anomaly-free custom model. The authors also propose a triplet contrastive anomaly inference strategy, incorporating the original/generated anomaly-free samples and text prompts. Extensive experiments are extensively conducted across 11 datasets.
strength_weakness: strength_weakness: 1. It is essentially a memory-augmented reconstruction-based anomaly detection (AD) method [1], which attempts to reconstruct the query image to its most similar anomaly-free counterpart. However, the reconstruction-based AD method also explores the Stable Diffusion (SD) denoising network [1]. Could you clarify the differences? If I understand correctly, the core difference is the inputs to SD, which are pairs of object text prompts and few-shot normal images.
2. Though Figure 4 demonstrates the effectiveness of the generated anomaly-free samples on three AD methods, how do these generated samples enhance the authors’ method? Can we use more or fewer generated normal samples instead of 100? An ablation study is required here.

[1] Gong et al. Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In ICCV, 2019.
[2] He et al. Diad: A diffusion-based framework for multi-class anomaly detection. In AAAI, 2024.
questions: questions: 1. Which data augmentations are used to augment few-shot normal images? Do these data augmentations vary across different categories?
2. It would be interesting to investigate how text prompts affect the anomaly-free customized model. For example, what if a specific category name (e.g., “cable”) is used to replace "object"?
3. How are \alpha and \beta determined in the final prediction? How about the ratio of the t-step?
4. It would be favorable to report F1-max, AP and PRO along with AUROC.
5. What about the one-shot setting?
limitations: limitations: The limitations are adequately addressed.
paper_summary: paper_summary: This paper addresses the issue of few-shot anomaly detection, which introduces an anomaly personalization method by using an anomaly-free customized generation model and performing a triplet contrastive anomaly inference strategy. Experiment evaluations across eleven datasets in three domains demonstrate its superior performance compared to the latest AD methods.
strength_weakness: strength_weakness: - There is a lack of discussion of the computation cost and inference speed of the proposed method since there are many generation steps. And the explicit description and exact number of prompts and images for generation are not clearly stated.

- The method introduces several hyperparameters (e.g., α, β) which require careful tuning. It's unclear how sensitive the results are to these hyperparameters and whether the paper provides sufficient guidance on setting them. Regarding the process of multi-level feature comparison, it would be better to clarify the effect of the number of multi-feature extraction blocks.

- The equations are not numbered and the line numbers are incomplete which makes it hard to reference.
questions: questions: 1. What are the exact components of $C_0$ and $\overline{C_0}$? As for the normal state prompts, is the number of $n$ below Line125 13? And what is the number of the memory bank $M$?

2. The "Triplet Contrastive Anomaly Inference" seems to work as a weighted prediction from three comparison aspects during testing. Where can it reflect the concept of "contrastive"? What is the full training objective?

3. Does $S_{text}$ reflect the degree of anomaly score? It seems that there are two types (both normal and abnormal) objects in the text prompts. How do they work in the same way as the equations of $S_{text}$ and $A_{score}$?

4. Is there any limitation for the proposed method to handle the open-vocabulary scenarios? And what is the computation cost and inference cost of the proposed method compared with other methods since there are many images for generation?
limitations: limitations: The authors provide a brief discussion of the limitations on zero-shot anomaly detection.
paper_summary: paper_summary: This paper proposes a new few-shot anomaly detection method based on one-to-normal personalization of query images using a diffusion model and a triplet contrastive inference process. It leverages a diffusion-based generation model to transform a query image into a personalized image towards the distribution of normal images, for use in the triplet contrastive inference. Experimental results on datasets from industrial, medical, and semantic domains demonstrate that the proposed method outperforms existing models.
strength_weakness: strength_weakness: - Details about the experimental settings are not fully provided, which limits reproducibility. For instance, there is no information on the selection of hyperparameters such as alpha and beta for A_score in the implementation details. It raises concerns about the model's performance. If alpha and beta were optimized based on test data, the claim of state-of-the-art performance is questionable. Details about the memory bank M such as its size and sensitivity of the performance to its hyperparameters are also missing. 
- The paper lacks theoretical analysis, relying solely on empirical evidence.
- Figure 1 does not provide sufficient information about the overall process. Particularly, it omits the training process of the diffusion model and the composition of the anomaly-free pool, which can cause confusion before reading the detailed explanations in the main text. 
- The definition of S_text in Sec 3.4 is unclear. It does not seem logical to apply the softmax function directly to the paired feature of F_q,l and F^T_text. 
- The lack of equation numbers in the Method section makes it difficult to follow. 
- There is no discussion of Table 3 in the paper.
questions: questions: - Can you provide visualizations or information about the distribution of the individual anomaly scores (S_n, S_p, S_text)?  It would be helpful to understand whether these scores focus on different parts or the same regions.
- The sensitivity of the model to text prompts should be further explored. Specifically, I would like to know the number of selected text prompts when generating personalized image and the rationale or justification for setting physical-level text prompts in the chosen template.
- In Table 4, using all the scores does not always correspond to the highest performance. A discussion on the possible reasons for this would be valuable. 
- Could you provide results from InCTRL in Figure 4, and also add lines or bars to compare each performance against the performance of the proposed model?
- Why is the performance on MVTec-AD different in Table 5 (96.8) and Table 4 (96.2)?

Minor issues:
- In Table 4, the top value for AFID is 85.2, not 84.7 as bold-faced.
- Tables 4 and 5 lack information on the few-shot setting.
- Typo: "semantice" should be "semantic" in the caption of Table 3.
limitations: limitations: - As mentioned in the paper, the method is not applicable in zero-shot scenarios.
- The theoretical exploration is lacking; there is only empirical evidence.
- The computational cost is likely to be high, which may not be suitable for real-world scenarios
- The approach depends on the performance of the generative model. And it is limited if the generated images are abnormal (potentially vulnerable to attacks).
paper_summary: paper_summary: This paper provides a novel method to tackle the issue of precision loss in more complex domains. They introduce an anomaly personalization method with a diffusion model to utilize the diffusions to obtain the normal sample distribution and exchange the anomaly image into normal ones. Finally, a triplet contrastive strategy is designed to obtain the score. Their approach obtains a SOTA result compared to the recent methods.
strength_weakness: strength_weakness: 1. Will the utilize of the diffusion model bring a long inference time, which may not be effective for real-time application?
2. Will the choice of the text influence the result? Is there any comparison result with different prompt choices?
questions: questions: See Weakness.
limitations: limitations: The authors have address the limitations in zero-shot scenarios.

Comments: title: Thank you for your feedback!
comment: We sincerely appreciate you taking the time to review our rebuttal and helpful feedback on our work. We are very glad that our response has addressed your concerns.
We sincerely appreciate you taking the time to review our rebuttal and for your positive feedback. We are very glad that our response has addressed your concerns.
Thank you for your feedback. I think your reply solve my concerns. I will keep my rate as Accept.
I highly appreciate the authors' helpful feedback. I agree with explanations on the differences between the diffusion model-based reconstruction and memory-based one. All other questions are well-solved through additional experiments so I would like to increase my rating and recommend a borderline acceptance.
***

>Q#4: It would be favorable to report F1-max, AP and PRO along with AUROC？

**A#4:** Following your suggestion, we have included the results of AUROC, F1-max,  AP, AUPRC, and PRO on MVTec dataset in the table below for a more comprehensive comparison, where our method consistently outperforms the baselines on all metrics.


|                |  Methods  |  AUROC  |  F1-max  |   AP   |  AUPRC  |  PRO   |
|:--------------:|:---------:|:-------:|:--------:|:------:|:-------:|:------:|
| **2-shot**     | WinCLIP   |  93.1   |   93.3   |  95.9  |  96.5   |  88.2  |
|                | InCTRL    |  94.0   |    -     |   -    |  96.9   |   -    |
|                | VAND      |  92.4   |   92.6   |   96   |    -    |  91.3  |
|                | Ours      | **95.1**| **94.3** |**96.5**|**97.3** |**92.1**|
| **4-shot**     | WinCLIP   |  94.0   |   93.5   |  96.2  |  96.8   |  88.5  |
|                | InCTRL    |  94.5   |  **-**   |   -    |  97.2   |   -    |
|                | VAND      |  92.8   |   92.8   |  96.3  |    -    |  91.8  |
|                | Ours      | **95.6**| **94.8** | **97** |**97.8** |**92.6**|
| **8-shot**     | WinCLIP   |  94.7   |   93.8   |  96.5  |  95.3   |  89.1  |
|                | InCTRL    |  95.3   |    -     |   -    |  97.7   |   -    |
|                | VAND      |  93.0   |   93.1   |  96.5  |    -    |  92.2  |
|                | Ours      | **96.2**| **95.1** |**97.4**|**98.9** |**93.1**|


***
For your convenience, we provide the experimental results mentioned in **the uploaded PDF** here to facilitate your review. We hope these supporting data address any concerns or questions you may have. If there is any confusion or if any part of our work requires further clarification, please do not hesitate to comment. We are more than willing to provide additional explanations and engage in further discussions.

***
>W#2: Can we use more or fewer generated normal samples instead of 100? An ablation study is required here.

**A#1:** Yes, we attempted to generate samples ranging from 10 to 300 to validate the effectiveness of the proposed method. We have placed the results here in the following table. Increasing the number of generated samples continues to enhance the results, likely because a larger sample set better represents the distribution of normal samples. However, for computational efficiency, we do not recommend generating too many samples (e.g., exceeding 200).

| Datasets | 10   | 30   | 50   | 100  | 150  | 200  | 300  |
|----------|------|------|------|------|------|------|------|
| MVTec-AD | 95.9 | 96.2 | 96.3 | 96.4 | 96.5 | 96.6 | 96.6 |
| RESC     | 94.7 | 95.2 | 95.4 | 95.6 | 95.6 | 95.7 | 95.9 |
| CIFAR    | 94.2 | 94.9 | 95.2 | 95.5 | 95.6 | 95.6 | 95.7 |

***

***
>Q#3: How are alpha and beta determined in the final prediction?

**A#2:** We set the parameters $\alpha$ and $\beta$ for $A_{score}$ to 1 and 0.5, respectively, this configuration remains consistent across all datasets. This choice was informed by our preliminary experiments, which demonstrated satisfactory performance across the majority of datasets under this setting. We provide more results in the following table below, demonstrating the robustness of our method to different choices of $\alpha$ and $\beta$.

| $\alpha$ | $\beta$ | MVTec | VisA | KSDD | AFID | ELPV | OCT2017 | BrainMRI | HeadCT | RESC | MNIST | CIFAR-10 | Average|
|:--------:|:-------:|:-----:|:----:|:----:|:----:|:----:|:-------:|:--------:|:------:|:----:|:-----:|:--------:|:--------:|
|    1     |   0.5   | 96.2  | 89.9 | 98.4 | 84.7 | 90.6 |  99.3   |   98.6   |  94.8  | 95.2 | 93.6  |   94.9   |   **94.2**   |
|   0.5    |    1    | 95.7  | 89.3 | 98.0 | 84.2 | 90.1 |  99.1   |   98.3   |  94.6  | 95.1 | 93.2  |   94.6   |   **93.9**   |
|    1     |    1    | 95.9  | 89.5 | 98.1 | 84.1 | 90.2 |  99.4   |   98.5   |  95.1  | 95.0 | 93.4  |   94.7   |   **94.0**   |
|   0.5    |   0.5   | 96.1  | 89.7 | 98.3 | 84.6 | 90.8 |  99.0   |   98.1   |  94.5  | 95.3 | 93.8  |   94.3   |   **94.1**   |
|    2     |    1    | 96.0  | 89.8 | 98.5 | 84.0 | 90.0 |  99.2   |   98.3   |  94.3  | 94.8 | 93.2  |   94.6   |   **93.9**   |
|    1     |    2    | 95.7  | 89.1 | 97.8 | 83.5 | 89.7 |  99.2   |   98.7   |  94.5  | 95.0 | 93.4  |   94.7   |   **93.8**   |
|    2     |    2    | 95.9  | 89.3 | 97.9 | 83.2 | 89.9 |  99.1   |   98.5   |  95.3  | 94.8 | 93.1  |   94.8   |   **93.8**   |


***

***

>Q#5: What about the one-shot setting?

**A#3:** In the one-shot setting, we have calculated the performance of our method and compared it with recent methods. The table below presents the AUROC comparison results, showing that our method maintains optimal performance on most datasets.

|                   | Datasets        | WinCLIP                   | InCTRL                   | Ours                       |
|:-----------------:|:---------------:|:-------------------------:|:------------------------:|:--------------------------:|
| **Industrial field** | **MVTec**       | 92.5±2.3                  | 93.2±1.7                 | **94.8±0.7**               |
|                   | **VisA**        | 83.6±2.5                  | 84.2±2.5                 | **87.0±1.7**               |
|                   | **KSDD**        | 94.0±0.5                  | **96.6±2.8**             | 96.5±1.6                   |
|                   | **AFID**        | 72.3±4.2                  | 76.0±3.2                 | **77.6±1.5**               |
|                   | **ELPV**        | 72.2±2.5                  | 82.8±1.2                 | **85.2±0.8**               |
| **Medical field**    | **OCT2017**     | 90.7±2.6                  | 93.0±2.3                 | **95.8±1.6**               |
|                   | **BrainMRI**    | 93.1±1.5                  | 96.7±2.4                 | **96.9±1.3**               |
|                   | **HeadCT**      | 91.7±1.8                  | 92.3±2.0                 | **93.7±1.2**               |
|                   | **RESC**        | 85.7±2.6                  | 87.6±2.9                 | **92.4±1.2**               |
| **Semantic field** | **MNIST**       | 76.3±1.7                  | 87.7±2.3                 | **91.8±0.6**               |
|                   | **CIFAR-10**    | 92.3±0.2                  | 93.2±0.9                 | **93.6±0.5**               |

***