Do LLMs Understand Visual Anomalies?  Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 06:50:40

Abstract: Large vision-language models (LVLMs) are markedly proficient in deriving visual representations guided by natural language.
Recent explorations have utilized LVLMs to tackle zero-shot visual anomaly detection (VAD) challenges by pairing images with textual descriptions indicative of normal and abnormal conditions, referred to as anomaly prompts. However, existing approaches depend on static anomaly prompts that are prone to cross-semantic ambiguity, and prioritize global image-level representations over crucial local pixel-level image-to-text alignment that is necessary for accurate anomaly localization. In this paper, we present ALFA, a training-free approach designed to address these challenges via a unified model. We propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. We further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on the challenging MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec AD and 8.9% on VisA compared to state-of-the-art zero-shot VAD approaches.

Main paper: 
 introduction: 
 Visual anomaly detection (VAD) has gained momentum in a wide spectrum of domains, including industrial quality control [2, 37] , video surveillance [14, 44] , medical diagnostics [10, 33, 53] and etc. This complex task involves both anomaly classification and localization for images, i.e., image-level and pixel-level anomaly detection. Inevitably, VAD faces two fundamental challenges due to the nature of its detection targets. First, the diversity of image objects makes the categories of anomalies a long-tail distribution [39] . To address the diverse range of images, a universal, category-agnostic model is required, as opposed to the traditional approach of deploying dedicated models for specific visual inspection tasks. The latter approach is unscalable and inefficient due to the long tail characteristic of the problem [26, 31] . Second, anomaly images are rare and have great variations [6, 19, 60] . In real-world applications like industrial VAD, collecting a sufficient and diverse training sample set is both costly and time-consuming. This scarcity complicates the training of traditional one-class or unsupervised VAD models, especially in cold-start scenarios [37] .The introduction of zero-shot methods offers a promising solution to these challenges. The emergence of large-scale models [23, 35] has revolutionized VAD profoundly. Recently, several large vision-language models (LVLMs) have been introduced for zeroshot VAD [4, 13, 17, 19, 58] . These works harness the exceptional generalization ability of LVLMs, pre-trained on millions of imagetext pairs, which showcase promising zero-shot performance in both seen and unseen objects. Nonetheless, due to the inherent lack of comprehensive information on data and the absence of explicit supervision, the zero-shot regime remains particularly challenging, with significant potential yet to be exploited compared to fully-supervised benchmarks.There are two major limitations. First, existing works rely on fixed textual descriptions of images, termed anomaly prompts, including both abnormal and normal prompts. In LVLM-based VAD, anomaly prompts elucidate the semantics of normalities and anomalies and guide the vision modules on how the two states are defined, the quality of which, therefore, plays a critical role in the zero-shot detection capability of LVLMs. The current practice of manually crafting prompts demands extensive domain expertise and considerable time, while also facing the challenge of cross-semantic ambiguity, which is illustrated in Figure 1 and will be discussed in depth in Sec. 4.2. This limitation calls for more informative and adaptive anomaly prompts. Second, although LVLMs, trained for image-text cross-modal alignment, can detect anomalies globally by aligning image-level representations with anomaly prompts, they face difficulties in localizing anomalies precisely, i.e., achieving pixel-level detection. Such local pixel-level alignment is central to zero-shot anomaly segmentation [19] .In this paper, we focus on zero-shot modeling and address the limitations of existing models with a proposal called ALFA -Adaptive LLM-empowered model for zero-shot visual anomaly detection with Fine-grained Alignment. We introduce a run-time prompt adaptation strategy to efficiently generate informative and adaptive anomaly prompts, which obviates the need for laborious expert creation and tackles cross-semantic ambiguity. Leveraging the zeroshot capabilities of an LLM [3] that is renowned for its proficient instruction-following abilities [24] , this strategy automatically generates a diverse range of informative anomaly prompts for VAD. Next, we present a contextual scoring mechanism to adaptively tailor a set of anomaly prompts for each query image. To fully excavate the local pixel-level semantics, we further propose a novel fine-grained aligner that generalizes the image-text alignment projection from global to local semantic space for precise anomaly localization. This cross-modal aligner enables ALFA to achieve global and local VAD within one unified model without requiring additional data or fine-tuning. We summarize our main contributions as follows:• We identify a previously unaddressed issue of cross-semantic ambiguity. In response, we present ALFA, an adaptive LLMempowered model for zero-shot VAD, effectively resolving this challenge without the need for extra data or fine-tuning. • We propose a run-time prompt adaptation strategy that effectively generates informative anomaly prompts and dynamically adapts a set of anomaly prompts on a per-image basis. • We develop a fine-grained aligner that learns global to local semantic space projection, and then, generalizes this projection to support precise pixel-level anomaly localization. • Our comprehensive experiments validate ALFA's capacity for zero-shot VAD across diverse datasets. Moreover, ALFA can be readily extended to the few-shot setting, which achieves stateof-the-art results that are on par or even outperform those of full-shot and fine-tuning-based methods. 
related work: 
 Vision-language modeling. Large Language Models (LLMs) such as GPT [3] and LLaMA [46] have achieved remarkable performance on NLP tasks. Since the introduction of CLIP [35] , large Visual-Language Models (LVLMs) like MiniGPT-4 [59] , BLIP-2 [27] , and PandaGPT [43] have shown promise across a range of languageguided tasks. Without additional fine-tuning, text prompts can be used to extract knowledge in the downstream image-related tasks such as zero-shot classification [32] , object detection [21] , and segmentation [49] . Consequently, LVLMs offer the potential to advance language-guided anomaly detection in a zero-shot manner.In this paper, we delve deeper into exploring how to optimize the utilization of LVLMs for visual anomaly detection (VAD).Visual anomaly detection. Given the scarcity of anomalies, conventional VAD approaches primarily focus on unsupervised or self-supervised methods relying exclusively on normal images. These approaches fall into two main categories: generative models [26, 31, 36, 50, 56] that utilize an encoder-decoder framework to minimize the reconstruction error, and feature embedding-based models that detect anomalies by discerning variations in feature distribution between normal and abnormal images. The latter includes one-class methods [30, 38, 47] , memory-based models [16, 34, 37] and knowledge distillation models [1, 12, 40, 52] hinging on the knowledge captured by networks pre-trained on large dataset.Recent research has delved into zero-shot VAD, reducing reliance on either normal or abnormal images and offering a unified anomaly detection model applicable across various image categories [4, 13, 17, 19, 28, 58, 61] . Notably, WinCLIP [19] pioneers the potential of language-driven zero-shot VAD, leveraging CLIP to extract and aggregate multi-scale image features. MuSc [28] proposes a looser zero-shot approach that utilizes a pre-trained Vision Transformer (ViT) [15] to extract patch-level features and assesses anomaly scores by comparing the similarity of patches between the query image and hundreds of unlabeled images. However, these approaches still suffer from several limitations, which require manual prompt crafting, intricate post-processing of extra data, and additional fine-tuning. In contrast, ALFA is a training-free model for zero-shot VAD, obviating the need for extra data or additional finetuning, and generates informative and adaptive prompts without costly manual design. Probing through visual prompt engineering. In VAD, prompts describe image content to assess anomaly levels by aligning with both normal and abnormal prompts. Traditional prompt engineering [25, 29, 42 ] that adjusts the model with learnable tokens is unsuitable due to data requirements. Existing efforts [1, 4, 8, 13, 17, 19, 45] typically hand craft numerous descriptions for detection, e.g., WinCLIP [19] using a compositional prompt ensemble and SAA [4] employing a prompt regularization strategy. However, these predefined-based approaches are inefficient and suboptimal. Recent studies have explored using LLMs to generate prompts for object recognition [21, 51] , potentially alleviating the challenge of inefficient prompt design. However, directly applying this approach to VAD tasks leads to cross-semantic ambiguity, caused by textual descriptions encompassing various aspects of an image, some of which may not be present or prominent in the query image. To avoid this, this paper proposes a run-time prompt adaptation strategy utilizing an LLM, coupled with a contextual scoring mechanism, to generate informative and adaptive prompts, which effectively addresses the cross-semantic issue.  
preliminary 3.1 visual anomaly detection: 
 Anomaly detection aims to detect data samples that deviate from the majority or exhibit unusual patterns. Particularly, this paper focuses on visual anomaly detection (VAD), the objectives of which are to (1) detect anomalies globally for images, and (2) localize anomalies for pixels of each image locally, as formulated below: Definition 1 (Visual anomaly detection). Given an image 𝑥 ∈ R 𝐻 ×𝑊 ×𝐶 , VAD aims to predict whether 𝑥 and all its individual pixel 𝑥 𝑖 𝑗 , are anomalous or not, where 0 ≤ 𝑖 < 𝐻 and 0 ≤ 𝑗 < 𝑊 .In this study, we seek to develop a category-agnostic VAD approach that exhibits generalizability across categories 𝑐 𝑖 ∈ C, allowing the model to readily adapt to new categories without model retraining or parameter fine-tuning. Formally, for ∀𝑥 ∈ 𝑐 𝑖 , VAD can be achieved by computing anomaly scores 𝑆 𝑖 , 𝑆 𝑝 = M (𝑥; Θ 𝑚 ) for both global image-level 𝑆 𝑖 ∈ [0, 1] and local pixel-level 𝑆 𝑝 ∈ [0, 1] 𝐻 ×𝑊 ×1 , using a detection model M (•) parameterized by Θ 𝑚 .LVLMs provide a unified representation for both vision and language modalities, leveraging contrastive learning-based [7] LVLMs can be adopted for zero-shot language-guided anomaly detection for images. For instance, given an image 𝑥 𝑗 , two predefined text templates, i.e., "a photo of a normal [𝑐 𝑖 ]" and "a photo of an abnormal [𝑐 𝑖 ]" and the extracted text tokens 𝑡 + and 𝑡 -correspondingly, anomaly detection is achieved by exploiting the visual information extracted by the image encoder and computing an anomaly score for category 𝑐 𝑖 :EQUATIONwhich basically measures the proximity of the image 𝑥 𝑗 to the abnormal text template of category 𝑐 𝑖 by tapping into the visionlanguage alignment capability of LVLMs. 
methodology 4.1 overview: 
 In this paper, we propose an LLM-empowered LVLM model ALFA for zero-shot VAD. As shown in Figure 2 , ALFA first introduces a run-time prompt (RTP) adaptation strategy to generate informative prompts and adaptively manage a collection of prompts on a per-image basis via a contextual scoring mechanism (see Sec. 4.2). Unlike conventional run-time adaptation approaches, which require fine-tuning their pre-trained models during inference, our strategy functions without the requirement for any parameter update. Furthermore, we present a training-free fine-grained aligner to bridge the cross-modal gap between global and local semantic spaces, enabling precise zero-shot anomaly localization (see Sec. 4.3).The quality of textual prompts significantly influences the zero-shot detection capabilities of LVLMs. Figure 3 provides a visual overview of our prompt generation and adaptation process, with more details elaborated below.Informative prompt generation. Staying in line with the prompt learning trend [22, 57] , we first employ general expert knowledge to initialize the contrastive-state prompts, unlocking LVLMs' knowledge guided by language. We design unified templates with specific contents to generate comprehensive prompts covering task-relevant concepts thoroughly, which contrasts with prior approaches that A normal image of carpet would show a smooth, even surface with a consistent pattern or texture.A normal image of carpet contains fibers that are evenly distributed and lie in the same direction.An abnormal image of carpet would look like a pattern of dirt or stains that is not uniform or expected.An abnormal image of carpet looks like a rug that is torn, frayed, or discolored. either manually define image descriptions or integrate complex multifaceted prompts [4, 13] . Next, we decompose the unified anomaly prompt into components as: "A {domain} image of a {state} {class} [with {specific details}]", encompassing domain, state, and optional specific details sections. Then, we can readily generate contrastive-state prompts T 𝐶𝑆 = {t + 𝑐𝑠 , t - 𝑐𝑠 } as the base anomaly detector, where t+ 𝑐𝑠 = {𝑡 + 𝑐𝑠,0 , • • • , 𝑡 + 𝑐𝑠,𝑛 + 𝑐𝑠 }, t - 𝑐𝑠 = {𝑡 - 𝑐𝑠,0 , • • • , 𝑡 - 𝑐𝑠,𝑛 - 𝑐𝑠 }. 𝑛 +𝑐𝑠 and 𝑛 - 𝑐𝑠 indicate the number of normal and abnormal prompts generated by the unified template.Recognizing the potential for domain gaps to introduce language ambiguity, especially with a generic prompt, base anomaly detector derived from the unified template falls short. LLMs are repositories of extensive world knowledge spanning diverse domains, serving as implicit knowledge bases that facilitate effortless natural language queries [9] . This knowledge includes visual descriptors, enabling LLMs to furnish insights into image features. To avoid the costly and non-scalable practice of manually crafting prompts using domain-specific knowledge, we efficiently tap into LLMs for more informative prompts. To this end, we design prompts to query an LLM, e.g., "How to identify an abnormal bottle in an image?". Using this approach, we can derive precise descriptions of a wide range of objects in normal and abnormal states as Remark. In dealing with the diverse and unpredictable nature of anomalies, language offers the essential information to discern defects from acceptable deviations. Building upon the insights from [32] , we can enhance interpretability in VAD decisions by leveraging the capabilities of LLMs. Specifically, LLM can be employed to produce feature descriptions regarding anomalies. These descriptions can be provided to the LVLM to compute the logarithmic probability of each description pertaining to the image query. By examining the descriptors with high scores, we can gain insights into the model's decision. More details are provided in Section 5.4.T 𝐺 = {t + 𝑔 , t - 𝑔 }, where t + 𝑔 = {𝑡 + 𝑔,0 , • • • , 𝑡 + 𝑔,𝑛 + 𝑔 }, t - 𝑔 = {𝑡 - 𝑔,0 , • • • , 𝑡 - 𝑔,𝑛Cross-semantic ambiguity. In an ideal scenario, LVLMs for zeroshot VAD should be capable of recognizing the close correlation between normal images and their respective normal prompts, while identifying a more distant association with abnormal prompts. The relative distances to normal and abnormal prompts are crucial for LVLMs to detect anomalies effectively. However, by visualizing the semantic space of LVLMs, we observed an overlap and intersection in the feature distributions of both normal and abnormal prompts, as depicted in Figure 4 (a). This leads to situations where the features of certain anomalous images are closer to normal prompts while being distant from certain abnormal prompts. We refer to this phenomenon as cross-semantic ambiguity. We attribute this phenomenon to the intricate nature of textual descriptions and the semantic correlation between text and image. This is exacerbated by prompts covering diverse aspects of the image, some of which might not be salient or even absent in certain images. Anomaly detection is thus susceptible to cross-semantic ambiguity poisoning. Therefore, there is a pressing need for an effective remedy to adaptively manage a set of normal and abnormal anomaly prompts corresponding to each query image without semantic overlap. Contextual scoring mechanism. To address the persistent challenge of the cross-semantic ambiguity in VAD, we propose a contextual scoring mechanism, which adaptively adjusts a set of anomaly prompts on a per-image basis.Specifically, given a query image 𝑥 ∈ R 𝐻 ×𝑊 ×𝐶 and the vanilla anomaly prompts T 𝑣𝑎𝑛𝑖𝑙𝑙𝑎 := {T 𝐶𝑆 , T 𝐺 } = {t + 𝑐𝑠 , t - 𝑐𝑠 , t + 𝑔 , t - 𝑔 }, their embeddings can be obtained using the pre-trained image and text encoders of LVLMs, denoted as 𝑓 (𝑥) ∈ R 𝑑 and 𝑔(𝑡) ∈ R 𝑑 , where 𝑡 ∈ T 𝑣𝑎𝑛𝑖𝑙𝑙𝑎 and 𝑑 denotes the dimension of the latent semantic space. We calculate the cosine similarity between the embeddings of the query image 𝑥 and normal {t + 𝑐𝑠 , t + 𝑔 } and abnormal prompts {t - 𝑐𝑠 , t - 𝑔 } respectively as:EQUATIONEQUATIONIdeally, the distances between images and prompts for normal and abnormal categories should fall into two non-overlapping intervals. Specifically, normal images should be closer to normal prompts, while their distance to abnormal prompts should be farther, and vice versa for abnormal images. However, in practice, considering the heterogeneous nature of textual descriptions, not all descriptions of normal or abnormal conditions can be observed in a single image, which leads to the presence of some redundant or even noisy prompts that could negatively impact the model's detection performance. In this regard, we formulate the contextual score as a logistic function [20] to quantify the prompt's impact in discerning abnormalities from normal occurrences. For each Uncovering LLM's Capabilities in Zero-shot Anomaly Detection MM '24, October 28-November 01, 2024, Melbourne, Australia prompt 𝑡 ∈ T 𝑣𝑎𝑛𝑖𝑙𝑙𝑎 , its contextual score is calculated as follows,EQUATIONwhere 𝑑 𝑥 (𝑡) represents the cosine similarity between the prompt 𝑡 and the query image 𝑥, and 𝑘 is an adjustment parameter that controls the slope of the scoring function. Empirically, we set 𝑘 to 1, ensuring the scoring function exhibits a moderate rate of change beyond the interval. D (•, •) is used to calculate the distance between a point and an interval as follows, D (𝑑 𝑥 (𝑡 ), 𝑑 + 𝑖 ) = max(0, max(min𝑖 {𝑑 + 𝑖 (𝑥 ) } -𝑑 𝑥 (𝑡 ), 𝑑 𝑥 (𝑡 ) -max 𝑖 {𝑑 + 𝑖 (𝑥 ) } ) ) D (𝑑 𝑥 (𝑡 ), 𝑑 - 𝑗 ) = max(0, max(min 𝑗 {𝑑 - 𝑗 (𝑥 ) } -𝑑 𝑥 (𝑡 ), 𝑑 𝑥 (𝑡 ) -max 𝑗 {𝑑 - 𝑗 (𝑥 ) } ) )The contextual score S 𝑐 (𝑡) of the prompt 𝑡 is constrained within the range[0, 1]. Considering the interval [min 𝑖 {𝑑 + 𝑖 (𝑥)}, max 𝑖 {𝑑 + 𝑖 (𝑥)}] and [min 𝑖 {𝑑 - 𝑖 (𝑥)}, max 𝑖 {𝑑 - 𝑖 (𝑥)}],when the distance between the prompt and the query image in the semantic space 𝑑 𝑥 (𝑡) places farther from another interval than the one it belongs to, the contextual score approaches 1, indicating a strong relevant, and vice versa. In cases where the distance 𝑑 𝑥 (𝑡) straddles both intervals, the score settles at 0, indicating an indeterminate relevance. Therefore, during inference, we employ the contextual scoring mechanism to filter out prompts with a contextual score of 0, retaining only those in non-overlapping intervals, represented as T := {T + , T -}, with T + and T -representing the normal and abnormal prompts, respectively.We outline the procedure of RTP adaptation in Algorithm 1, and visualize the feature distribution of prompts processed through the contextual scoring mechanism in Figure 4 (b), which demonstrates that the proposed contextual score effectively addressed the cross-semantic ambiguity. Notably, the anomaly prompt T varies depending on the specific query image, which aligns with the intuitive notion that prompts and their numbers tailored to different object classes should naturally differ. Even within the same class, different images necessitate different emphases on individual prompts. Consequently, the implementation of the contextual scoring mechanism offers an adaptive approach to managing a set of prompts on a per-image basis, which enables the selected prompts that are better suited to the unique characteristics of each query image, thus enhancing the overall effectiveness of anomaly detection.Since anomaly localization requires predicting anomalies at the pixel-level, acquiring dense visual features is necessary. However, LVLMs enforce cross-modal alignment globally for images and text, creating a cross-modal gap between the global prompt embeddings and local patch token embeddings. WinCLIP [19] attempts to address this issue by employing a sliding window to generate patch embeddings in a manner that simulates processing the global image instead of using patch-wise embeddings from the penultimate feature map. However, the localized patch may not encompass the description of the global image in the text prompt, leading to suboptimal performance. While AnomalyGPT [17] achieves alignment by generating pseudo-anomaly samples and introducing additional training, which is operationally intricate and lacks efficiency. Consequently, we propose a training-free fine-grained aligner to explicitly model the mapping between global and local semantic spaces. Calculate the contextual score S 𝑐 (𝑡 ) using Eq.( 4) 6:if 𝑆 𝑐 (𝑡 )>0 then 7:Add 𝑡 into T 8:end if 9: end for 10: return Anomaly prompts T Given a query image 𝑥 and its corresponding anomaly prompts T , their embeddings can be denoted as 𝑓 (𝑥) ∈ R 𝑑 and 𝑔(𝑡) ∈ R 𝑑 , where 𝑡 ∈ T and 𝑑 denotes the dimension of the latent space. Mathematically, the encoder architecture consists of vision transformers (ViT) [15] based on multi-head self-attention (MHSA) and a feedforward network (FFN) with layer normalization (LN) and residual connections that can be expressed as:EQUATIONEQUATIONwhere MHSA can be further formulated as:EQUATION𝑧 𝑙,𝑚 = softmax(EQUATIONEQUATION)where𝑧 0 = [𝑣, 𝑝 1 , • • • , 𝑝 𝑁 ],𝑣 represents the [CLS] token and 𝑝 1 , • • • , 𝑝 𝑁 are the patch tokens of the query image 𝑥 with a resolution 𝐻 × 𝑊 , and 𝑀 denotes the number of attention heads. In image processing, the Query-Key retrieval pattern at the final layer can be conceptualized as a type of global average pooling mechanism for capturing visual global descriptions. Concurrently, the Value component serves to furnish comprehensive information regarding each position or region within the image. In the current task, our aim is to delve into the interplay between global and local semantic information. Therefore, by adjusting the configuration of the Value, whose ensemble forms the output of the final attention mechanism, we can achieve a more nuanced handling of global and local information by the model. Consequently, the model gains the capability to discern the intricate correlation between global and local in a more adaptable manner.Specifically, for a dense visual input 𝑥 𝑖 𝑗 = 𝑥 ⊙ 𝑚 𝑖 𝑗 , where 𝑚 𝑖 𝑗 ∈ {0, 1}EQUATIONLikewise, we can implement multi-scale masked images to generate multi-scale visual embeddings paired with corresponding prompt embeddings. Using these, we calculate multi-scale anomaly maps and average them through harmonic averaging [19] for anomaly localization of a given query. Relying on the premise that an image can be classified as anomalous upon the detection of a single anomalous patch, the anomaly score for the image query is determined by combining the classification score in Eq. 10 with the maximum value of averaged multi-scale anomaly map as follow:EQUATIONOur ALFA adeptly accommodates few-shot scenarios by employing a memory bank to store patch-level features from normal samples, illustrated in Figure 2 . Anomaly localization is subsequently improved on top of 𝑆 𝐿 by calculating distances between query patches and their nearest counterparts in the memory bank.  
experiments: 
 In this section, we systematically evaluate ALFA for image-level and pixel-level anomaly detection through quantitative and qualitative analyses on various benchmarks. Ablation studies and explainable VAD results are also presented. Further implementation details and comprehensive experimental results are provided in the Appendix.Datasets. Our experiments are based on MVTec [2] and VisA [62] benchmarks, both containing high-resolution images with full pixellevel annotations. MVTec includes data for 10 single objects and 5 textures, while VisA includes data for 12 single or multiple object types. As our framework is entirely training-free, we exclusively utilize the test datasets for evaluation.Metrics. We use Area Under the Receiver Operating Characteristic (AUROC), Area Under the Precision-Recall curve (AUPR), and F1-score at optimal threshold (F1-max) as image-level anomaly detection metrics. Besides, we report pixel-wise AUROC (pAUROC), Per-Region Overlap (PRO) scores, and pixel-wise F1-max (pF1-max) in a similar manner to evaluate anomaly localization. Implementation details. We employ the OpenCLIP implementation [18] and its publicly available pre-trained models. Specifically, we use the LAION-400M [41] -based CLIP with ViT-B/16+ as our foundational model and GPT-3.5 (gpt-3.5-turbo-instruct) for anomaly prompt generation. Further setup details and the prompts list considered in this paper are available in the Appendix. In Table 1 , we compare ALFA with prior arts on MVTec and VisA benchmarks for both image-level and pixel-level zero-shot anomaly detection. Specifically, we compare ALFA with CLIP-AC [35] for image-level anomaly detection, Trans-MM [5] for pixel-level anomaly detection, and WinCLIP [19] and AnoVL [13] for both image-level and pixel-level anomaly detection. For fairness, we use AnoVL - [13] for comparison, representing AnoVL without fine-tuning and data augmentation, while the comparison with the complete AnoVL is presented in Sec. 5.6. More details about the baselines are available in the Appendix A. For both image-level and pixel-level VAD, ALFA demonstrates significant improvements over all baselines across all metrics on both benchmarks. Notably, compared to the runner-up, we achieve a 12.1% enhancement in PRO for pixel-level anomaly detection on MVTec and a 8.9% improvement on VisA. Similarly, for image-level anomaly detection, we outperform the suboptimal method by 1.5% on MVTec and by 4.0% on VisA in terms of AUROC. A detailed breakdown of these gains is presented in Section 5.3 through ablation studies.Qualitative results. In Figure 5 , qualitative results for different objects with various anomalies are showcased. In all instances, ALFA yields an anomaly map that exhibits greater concentration on the ground truth compared to previous methods, aligning with the findings from the quantitative results. Subtle, ALFA fares better under various sizes and quantities of anomalies, demonstrating its versatility. More visualizations can be found in the Appendix C.Component-wise analysis. Ablation studies on key ALFA modules, including RTP adaptation and the fine-grained aligner, demonstrate their significant contributions to overall detection performance, detailed in Table 2 . Employing a one-class design using only normal prompts as a baseline in the first row, we emphasize the significance of template-based and LLM-based prompt generator in capturing various anomalous patterns. The contextual scoring mechanism, denoted as S 𝑐 in Table 2 , further enhances performance by adaptively managing the anomaly prompts customized for each query image without cross-semantic issue. Furthermore, the finegrained aligner proves to be a crucial contributor, especially in enhancing pixel-level anomaly detection in zero-shot scenarios.Analysis on anomaly prompt. In Table 3 , we demonstrate that ALFA achieves superior detection performance while significantly reducing human efforts in designing prompts. We present the number of prompts per label employed by each method. In general, LVLMs tend to exhibit improved performance with an increase in the number of prompts. However, when cross-semantic ambiguity limits the effectiveness of prompts, increasing their number may not necessarily lead to performance improvement, as evidenced by the results of AnoVL and WinCLIP as shown in Table 3 . In ALFA, the range of prompts per label on each class spans from 88 to 216, with an average of 146. Notably, we only design 72 prompts based on the template, a notable 53.2% reduction as compared to over 150 required by baselines, for better detection results, showcasing ALFA's ability to effectively tackle the cross-semantic issue and unlock the full potential of language for zero-shot VAD. We also assess the effect of varying numbers of prompts from template-based and LLM-based generator, with additional results in Appendix B. We further assess ALFA using GPT-3 (text-davinci-002) and GPT-3.5 (gpt-3.5-turbo-instruct) for automatic prompt generation, observing superior performance with GPT-3.5. To examine the scalability of ALFA, we also conduct tests on various CLIP backbones, with detailed results provided in the Appendix B. Moreover, by augmenting the input query of the GPT-3.5 as "state the description beginning with: An abnormal/normal image of {class}", the resulting prompts are formulated to preserve syntactic consistency to the greatest extent possible, aligning with the text in CLIP pre-training dataset. This augmentation contributes to further improved results.We present results for explainable anomaly detection in Figure 6 , where the bars illustrate the descriptor similarity to the image predicted as an anomaly in the CLIP latent space. Concretely, we condition descriptors on the class name by prompting the language model with the input: "Q: What are useful descriptions for distinguishing an anomaly {class} in a photo? A: There are several key descriptions to tell there is an anomaly {class} in a photo: -" where "-" is used to generate point-by-point characterizations as descriptors. Figure 6 shows the top five descriptors that emerge from GPT-3.5, encompassing colors, shapes, and object parts for both class-specific and class-agnostic descriptions. These descriptions enable ALFA to look at cues easily recognizable by humans, enhancing interpretability for decision-making in VAD tasks.We expand the capabilities of ALFA to include the few-shot setting, allowing for enhanced performance across scenarios with limited data. We report the mean and standard deviation over 5 random seeds for each measurement in Table 4 and Table 5 . We benchmark ALFA against PatchCore [37] and WinCLIP [19] . PatchCore utilizes few-shot images for generating nominal information in its memory bank, and the full-shot version of PatchCore will be discussed in Section 5.6. In this setting, ALFA consistently outperforms all baselines across all metrics, highlighting the efficacy of language prompts and multi-modal alignment for VAD. Moreover, with an increase in the shot number, ALFA exhibits improved performance, emphasizing the synergy between language-driven and reference normal image-based models.As our anomaly score and anomaly map are dual-composite, we conduct further ablation studies on their distinct components, detailed in Table 6 . For image-level VAD, as the number of shots increases, the significance of 𝑆 𝐺 in anomaly score gradually becomes evident, as it allows the introduction of information from normal images in the memory bank to serve as supervision for VAD. Meanwhile, max 𝑆 𝐿 consistently brings further performance improvement based on 𝑆 𝐺 . For pixel-level VAD, we assess the impact of image features generated by masks of various scales, using patches as the unit and a patch size of 16×16 in our foundational model. We also report the average inference time per image across different few-shot settings, evaluated on a server with Xeon(R) Silver 4214R CPU @ 2.40GHz (12 cores), 128G memory, and GeForce RTX 3090. We find that integrating image features at different scales notably improves performance by incorporating local information. However, scaling up the size comes at the cost of increased computational demands, impacting inference speed. Thus, we opt for a scale range of [2, 3] to achieve an optimal trade-off between inference speed and performance.We benchmark ALFA against prominent unsupervised and finetunerequired VAD methods in a unified setting for fairness. Most baselines undergo training or fine-tuning on normal samples encompassing all classes within the dataset. Additionally, we include three full/many-shot training-free methods, PatchCore [37] , SAA+ [4] and MuSc [28] . As depicted in Table 7 , our zero-shot ALFA is competitive with the baselines that require more information whether in the form of additional normal samples or training. In the 4-shot scenario, ALFA surpasses most baselines, underscoring the complementary roles between language and vision in VAD. 
conclusions: 
 In this paper, we present an adaptive LLM-empowered model ALFA that focuses on vision-language synergy for VAD. Capitalizing on the robust zero-shot capabilities of LLMs, the proposed runtime prompt adaptation strategy effectively generates informative prompts by tapping into the vast world knowledge encoded in their billion-scale parameters. This adaptation strategy is complemented by a contextual scoring mechanism, ensuring per-image adaptability while mitigating the cross-semantic ambiguity. Additionally, the introduction of a novel training-free fine-grained aligner further bolsters ALFA, generalizing the alignment projection seamlessly from the global to the local level for precise anomaly localization.Experimental results demonstrate ALFA's superiority over existing zero-shot VAD approaches, providing valuable interpretability.  
 
 Appendix:  


Decision: The paper initially received positive indications (2BA, 2WA), which were later raised after the rebuttal phase (3WA final ratings + 1 preliminary BA not uploading the final rating). Some concerns raised during the initial reviews have been cleared during the rebuttal, so the paper can be accepted. Considering the ratings of the paper, the quality of the reviews, and the quality of the paper perceived by the AC, the paper can also be accepted as oral.

Reviews: review: paper_summary: paper_summary: This manuscript proposes a novel adaptive LLM-empowered model ALFA for zero-shot anomaly detection, which focuses on vision-language synergy. Specifically, this method proposes a prompt adaptation strategy to mitigate the cross-semantic ambiguity by a contextual scoring mechanism. Furthermore, the method introduces a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly location. The experiment results demonstrate the superiority of the proposed framework to some extent.
strength_weakness: strength_weakness: 1. This paper designs modules from the perspective of cross-modal fine-grained alignment, which is innovative in the field of zero-shot anomaly detection.
2. The proposed method is straightforward and has theoretical feasibility. Meanwhile, this paper conducts experiments from multiple perspectives to prove the effectiveness of the proposed method compared with other SOTA methods.
3. Theoretically, all VLM-based anomaly detection models can be fine-grained aligned based on the idea of this paper, which has a relatively high application value.
limitations: limitations: 1. The component-wise ablation experiments do not seem to be complete and are not performed for each case individually. It would be better to complete the ablation experiments.
2. The visualization results of different methods are compared in Fig. 5, however, the precise localization ability of the proposed method is not reflected. It is desirable to reflect the performance difference in anomaly localization before and after using the aligner through some visualization results.
paper_summary: paper_summary: The paper proposes a training-free zero-shot visual anomaly detection model called ALFA that aims to leverage the capabilities of language models to improve zero-shot anomaly detection performance. This strategy utilizes a contextual scoring mechanism for adaptive prompt adaptation and semantic disambiguation for each image. Additionally, a novel fine-grained aligner is introduced, which projects the image-text alignment from global semantic space to local semantic space, enabling precise anomaly localization at the pixel level.
strength_weakness: strength_weakness: 1. The article proposes the ALFA model as a training-free zero-shot visual anomaly detection method. ALFA generates adaptive anomaly prompts and achieves precise pixel-level anomaly localization. It leverages a large language model to generate informative anomaly prompts and adapts them on a per-image basis using a contextual scoring mechanism.
2.The ALFA model introduces a fine-grained aligner that generalizes the projection from global to local semantic space, enabling global and local zero-shot anomaly detection without requiring additional data or fine-tuning.
3.The proposed approach is evaluated comprehensively on MVTec and VisA benchmarks, demonstrating the effectiveness of the ALFA model in zero-shot anomaly detection. Furthermore, the ALFA model can be easily extended to few-shot settings, achieving state-of-the-art results that are on par with or even surpass full-shot and fine-tuning-based methods.
limitations: limitations: （1）Word Choice and Grammar Corrections:
In line 88, the phrase “and etc” should be revised to simply “etc.” as “etc.” already implies the continuation of a list.
The spelling of “run-time” in the description of Figure 2 on line 250 appears to be incorrect. It should be unified to “runtime” throughout the document.
There seems to be a grammatical inconsistency in line 723 with the phrase “in terms of AUROC.” It would be more accurate to revise it to “in AUROC.”
In line 765, the term ‘VisA’ appears to be misspelled as ‘Visa’. It is crucial to ensure the correct capitalization is used throughout the document for consistency and accuracy.”
（2）Insufficient Explanation of Model Utilization:
The paper could benefit from a more detailed explanation of how the model leverages a memory bank to improve defect localization in scenarios with few-shot.
（3）Questions Regarding Model Generalizability:
Since the evaluation dataset employed is industrial, it raises the question of whether the model is solely capable of detecting defects in industrial settings or if it can be generalized to detect defects in other sectors as well.
（4）Model Limitations and Future Work:
It would be constructive for the manuscript to discuss any limitations of the model and suggest avenues for future research and development.
paper_summary: paper_summary: This paper addresses zero-shot image-level anomaly detection (AD) with LLM's anomaly description generation capability. Motivated by the weakness of class-conditional anomaly description generation when image context is unavailable, this paper proposes two components, i.e., run-time anomaly prompt scoring mechanism and fine-grained aligner, which reduces the ambiguity of anomaly text description. Experiments verify the superiority of the proposed method for zero-shot AD.
strength_weakness: strength_weakness: Overall, the paper is of good quality.
The main strengths are that:
(1) The motivation to address the ambiguity of anomaly description when image context is not used is clear.
(2) Accordingly, this paper proposes several components to adapt the high-level representation/knowledge from frozen LLM to the zero-shot fine-grained AD task.
(3) The proposed method is verified to be effective.
limitations: limitations: I have only a few minor comments on the experimental part.
(1) Although this method tries to make the prompt generation process more automatic, the proposed method still requires a large number of anomaly descriptions through manual designs. How much does the accuracy and number of anomaly descriptions affect the performance? 
(2) It would be helpful to provide the computational cost, as some people in the AD community would emphasise the practical applicability of AD methods.
(3) It would be interesting to test the upper bound of the proposed method with much more powerful LLMs such as GPT-4v. That is, if the LLM is much more powerful, would this resolve/reduce the ambiguity of the anomaly description?
paper_summary: paper_summary: The authors employ the Large vision-language models to deal with the problem of zero-shot VAD. They propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. They further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on the challenging MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec AD and 8.9% on VisA compared to state-of-the-art zero-shot VAD approaches.
strength_weakness: strength_weakness: The method is simple and easy to understand, the motivation is clear, the logical structure is reasonable, and it leverages the advantages of popular large vision-language models. The problem addressed is highly practical.
limitations: limitations: 1. What are the shortcomings of your method?
2. Tables 4 and 5 need to include comparisons with more recent methods.
3. More experimental analysis is needed to demonstrate the effectiveness of the method.