SimCLIP: Refining Image-Text Alignment with Simple Prompts for Zero-/Few-shot Anomaly Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 18:11:17

Abstract: Recently, large pre-trained vision-language models, such as CLIP, have demonstrated significant potential in zero-/few-shot anomaly detection tasks. However, existing methods not only rely on expert knowledge to manually craft extensive text prompts but also suffer from a misalignment of high-level language features with fine-level vision features in anomaly segmentation tasks. In this paper, we propose a method, named SimCLIP, which focuses on refining the aforementioned misalignment problem through bidirectional adaptation of both Multi-Hierarchy Vision Adapter (MHVA) and Implicit Prompt Tuning (IPT). In this way, our approach requires only a simple binary prompt to accomplish anomaly classification and segmentation tasks in zero-shot scenarios efficiently. Furthermore, we introduce its few-shot extension, SimCLIP+, integrating the relational information among vision embedding and skillfully merging the cross-modal synergy information between vision and language to address AD tasks. Extensive experiments on two challenging datasets prove the more remarkable generalization capacity of our method compared to the current state-of-the-art.

Main paper: 
 related work 2.1 vision-language models: 
 The Vision-Language models have attracted significant interest in the field of computer vision through the integration of language supervision with images. Recent V-L models [18, 30] employ joint learning of two branches to connect these two modalities. They further align both the vision and language in the multi-modal feature space. Compared to models solely trained through vision supervision, these V-L models encode richer multi-modal representations.During the training process, these models can more effectively comprehend the natural world by leveraging information from both modalities, contributing to their outstanding performance across a wide range of tasks, including those involving zero-shot or few-shot visual recognition tasks. While these pre-trained V-L models learn generalized representations, effectively fine-tuning them to adapt to specific downstream tasks remains a challenging issue. Many studies have achieved state-of-the-art performance in downstream tasks, including image recognition [1, 26] , object detection [12, 29, 36, 41] , and segmentation [4, 9, 27, 32, 42, 48] , by employing customized approaches to adapt V-L models for these specific tasks.Prompt learning originated in the field of Natural Language Processing (NLP). Large Language Models (LLMs) [3, 7, 8, 22, 38, 46] acquire extensive knowledge during the pre-training phase by employing self-supervised learning methods on massive corpora. Prompt learning [10, 15, 21, 23, 34, 39] was developed to bridge the gap between Pre-trained Language Models (PLMs) and specified downstream tasks. For example, when recognizing the emotion of a social media post, "I got fired today," we may continue with a prompt "I felt so [blank]" and ask the PLMs to fill the blank with an emotion-bearing word. In this way, prompt learning may close the gap between PLMs and downstream tasks by converting downstream tasks into fillin-blank tasks. Inspired by prompt learning in NLP, the paradigm of prompt learning has recently gradually expanded to encompass other domains, including V-L models [11, 19, 20, 45, 50] . In this paper, we explore a crucial issue: how to mitigate the gap between the CLIP [30] and AD tasks with prompt learning.Recent anomaly detection research has primarily concentrated on addressing anomaly classification and localization tasks with an extremely limited number of normal images. RegAD [16] employs image alignment tasks to train a category-agnostic anomaly detection model. WinCLIP [17] needs the hand-crafted design of textual prompts for both normal and anomalous conditions, and anomalies are identified by calculating the cosine similarity between textual prompts and images. While WinCLIP exhibits excellent performance, its effectiveness is heavily influenced by the carefully hand-crafted textual prompts. In a practical industrial setting, the meticulous design of textual prompts for each category is a tedious and time-consuming task. AnomalyGPT [13] takes image features as input for the LLM, leveraging the knowledge of the LLM to capture anomalous patterns within the images. Due to the core reliance on LLM, AnomalyGPT introduces a considerable computational overhead during the inference stage, which is unfavorable for real-time applications. 
method: 
 In this section, we begin with a brief overview of CLIP [30] , which serves as the foundational model employed in our method. Subsequently, we elaborate on SimCLIP, employing a multi-hierarchy vision adapter (MHVA) and implicit prompt tuning (IPT) to achieve realignment between vision and language. Finally, we propose an extended version of SImCLIP, named SimCLIP+, to address few-shot AD tasks by integrating a feature-driven method and a prior-aware optimization algorithm.CLIP consists of two branches: the vision branch I is dedicated to capturing the visual features of images, and the language branch T converts text prompts into semantic embeddings. During the pre-training phase, CLIP employs contrastive learning [5, 14] to maximize the cosine similarity between vision and language features that correspond to the same semantics within the multi-modal feature space. This is intended to promote the alignment between visual and textual features. Thus, CLIP can effectively leverage the distance between two modalities to accomplish zero-shot recognition tasks. Let 𝑥 and {𝑃 𝑗 } 𝐾 𝑗=1 denote the inputs to the vision branch I and the language branch T respectively. Each prompt 𝑃 𝑗 corresponds to a category, with 𝐾 being the total number of categories assumed. Specifically, each 𝑃 𝑗 is obtained through a specific prompt template, such as "a photo of a {class}, " where the "{class}" token is replaced with the name of the 𝑗-th class. The prediction probability is then calculated as:EQUATIONwhere sim(•, •) denotes cosine similarity and 𝜏 is a temperature coefficient.In this section, we introduce SimCLIP, a method designed to efficiently drive CLIP to perform zero-shot AD tasks by leveraging a combination of the multi-hierarchy vision adapter and implicit prompt tuning. This approach aims to realign the vision and language modality within the multi-modal feature space. CLIP focuses exclusively on achieving precise alignment between high-level global semantic information extracted from images and text during the pretraining phase. However, the extraction of highlevel global semantic information often results in the loss of spatial details, thereby impeding the accurate localization of anomalies within images.To overcome this limitation, as shown in Figure 3 , we extract patch-level feature maps at various hierarchies. Formally, let 𝐿 represent the subset containing the indexes of the hierarchies to be utilized. The feature maps of the 𝑙 ∈ 𝐿 are represented as F 𝑙,𝑖 ∼ I 𝑙 (𝑥 𝑖 ) ∈ R 𝐻 𝑙 ×𝑊 𝑙 ×𝐶 𝑙 , where 𝑥 𝑖 ∈ D 𝑡𝑟𝑎𝑖𝑛 ∪ D 𝑡𝑒𝑠𝑡 denotes the input image. Here 𝐻 𝑙 , 𝑊 𝑙 , and 𝐶 𝑙 refer to the height, width, and depth of the feature map, respectively. We then select a feature slice at the spatial position ℎ ∈ {1, 2, . . . , 𝐻 𝑙 } and 𝑤 ∈ {1, 2, . . . ,𝑊 𝑙 } from the 𝑙-th layer's feature map, denoted by F 𝑙,𝑖 (ℎ, 𝑤) ∼ I 𝑙 (𝑥 𝑖 , ℎ, 𝑤) ∈ R 𝐶 𝑙 , which is a 𝐶 𝑙 -dimensional real vector. To enable cross-modal interaction with text prompts and retain spatial information within the patch-level feature maps from different levels, we introduce a multi-hierarchy vision adapter (MHVA) A 𝑙,𝜃 . The adaptation process is represented as follows:F ′ 𝑙,𝑖 = A 𝑙,𝜃 (F 𝑙,𝑖 (ℎ, 𝑤)|F 𝑙,𝑖 (ℎ, 𝑤) ∈ I 𝑙 (𝑥 𝑖 )), where 𝑙 ∈ 𝐿. (2)We aim to maintain a design for the adapter that is both simple and efficient. Therefore, a linear probe is employed to construct the multi-hierarchy vision adapter. One choice for locating anomalies in an image is to calculate the feature similarity between the local regions of the input image (according to Eq.( 2)) and text prompts. This is done in both [17] and [6] undoubtedly introduces the following two problems. Firstly, the abstract features extracted by the CLIP language branch are more tailored toward natural image classification tasks and hold limited relevance to segmentation tasks in industrial anomaly detection, with minimal overlap. Secondly, there is an inherent lack of alignment between the fine-level vision features and high-level language features. Directly utilizing these features would significantly degrade the model's generalization capability. In summary, this unidirectional vision adaptation method fails to exploit the potential of CLIP in AS tasks. Inspired by these insights, in addition to adapting the vision modality using the multi-hierarchy vision adapter, we also introduce implicit prompt tuning (IPT) to refine the language modality. Firstly, we design simple binary prompts representing 'normal images' and 'anomalous images', respectively. In the remainder of this paper, unless explicitly indicated otherwise, the following binary prompts are the default:EQUATIONHere, the symbol [𝑐𝑙𝑠] denotes the category's name. Due to the limited capacity of text information, for instance, natural language is insufficient to encompass the entirety of anomaly types. Furthermore, to diminish reliance on hand-crafted prompts, we employ prompt learning to refine language features in latent embedding space. Let 𝑇 𝑙𝑎𝑠𝑡,𝑗 = T (𝑃 𝑗 ) ∈ R 𝑁 ×𝐶 𝑙𝑎𝑠𝑡 denotes the output of the last transformer layer of the language branch, where 𝑗 ∈ {0, 1} and 𝑁 denotes the number of tokens. As shown in Figure 3 , we employ an implicit prompt tuning L 𝜃 integrated with learnable task-specific prompt embeddings P tailored for AD tasks to refine the original binary prompt. According to [24] , it is evident that directly updating the P can result in unstable optimization processes and a slight performance decline. Therefore, we randomly initialize a relatively small matrix P 𝑠 , then pass it through a single multi-layer perceptron (MLP) layer to address this limitation. The final task-specific prompt embedding can be obtained as follows:EQUATIONAs shown in Figure 3 , the IPT L 𝜃 accepts prompt feature 𝑇 𝑙𝑎𝑠𝑡,𝑗 and task-specific prompt embedding P as inputs, consisting of two core components. Firstly, the weight transformation matrix𝑊 = {𝑤 𝑖 } 𝑁 𝑖=1 accurately preserves key features from the original prompt while eliminating redundant information, where each weight 𝑤 𝑖 ∈ [0, 1]. Subsequently, the feedforward network (FFN) skillfully integrates the key features with task-specific prompt embeddings to attain refined prompts. The process of refining the text prompt through IPT is denoted as:L 𝜃 (𝑇 𝑙𝑎𝑠𝑡,𝑗 , P) = FFN([𝑇 ⊤ 𝑙𝑎𝑠𝑡,𝑗 𝑊 , P]), where 𝑗 ∈ {0, 1}, (5) where P represents a new learnable task-specific prompt embedding in the latent feature space, and [•, •] signifies the concatenation operation. Compared to other prompt learning methods [49, 50] , IPT directly refines text prompts within the embedding space, expanding the solution space and markedly enhancing the flexibility of the learning process. Through the bidirectional adaptation facilitated by both MHVA and IPT for realigning both vision and language, the resulting anomaly segmentation outcome for language-driven at the 𝑙-th hierarchy can be generated as follows:EQUATIONwhere 𝑖 denotes the 𝑖-th input image and F ′ 𝑙,𝑖 represents the vision adaptation feature after the multi-hierarchy vision adapter. The final zero-shot AS result for SimCLIP is obtained by aggregating anomaly segmentation maps from different hierarchical levels, denoted as 𝑙 ∈𝐿 M 𝑙,𝑧𝑒𝑟𝑜 (F ′ 𝑙,𝑖 ). The zero-shot AC task can be accomplished by calculating the similarity between the vision features generated by the class token and the refined binary prompt.Notes. In this paper, we finetune the pre-trained CLIP using the test set from a publicly available dataset (e.g., MVTec-AD) and then evaluate the performance using another dataset. All parameters of CLIP are frozen except for the projection head of both two branches. Additionally, parameter updates are performed on the newly introduced multi-hierarchy vision adapter, the task-specific prompt embedding, and the IPT module during the finetune process. We employ cross-entropy loss for AC, while focal loss [25] and dice loss [28] are utilized for AS tasks.In this section, SimCLIP+ is proposed to tackle anomaly detection under limited samples, which leverages correlation information among vision embeddings and integrates the cross-modal synergy information generated by SimCLIP. Otherwise, SimCLIP+ also uses a prior-aware optimization algorithm to optimize the cross-modal synergy information mentioned above to further enhance the performance.We address the challenge posed by limited normal samplesD + 𝑁 = {𝑥 𝑖 } 𝑁 𝑖=1, where 𝑁 represents the number of normal images. Prior works [31, 37, 43, 44, 47] have extensively demonstrated the exceptional capability of CLIP in extracting features by vision branch. Motivated by these findings, we have decided to identify potential anomalies by measuring the distance at the embedding between the query and normal comparison images 𝑥 ∈ D + 𝑁 . Building upon Section 3.2, where spatial information from various hierarchies is utilized, we introduce a Memory Gallery module M 𝑙 across different levels. The M 𝑙 aims to store patch-level features of all comparison images extracted from the 𝑙-th hierarchy on the vision branch, where 𝑙 ∈ 𝐿 same to Section 3.2. As shown in Figure 3 , leveraging the embedding distance between the query and comparison images at patch-level on various hierarchies, the anomaly segmentation result M 𝑣 generated using vision embedding information is represented as follows:EQUATIONHere, F 𝑙,𝑖 (ℎ, 𝑤) denotes the feature slice of the 𝑖-th query image at 𝑙-th hierarchy. The feature-driven method leverages correlation information among vision embeddings and integrates the crossmodal synergy the information generated by SimCLIP. However, it overlooks the prior information between the refined text prompt and the normal comparison images. Inspired by these, we propose a prior-aware optimization algorithm, which is designed to further optimize the cross-modal synergy information mentioned above.  
experiments: 
 We conducted a series of experiments to evaluate the effectiveness of SimCLIP/SimCLIP+ in industrial anomaly classification and anomaly segmentation tasks. In addition, a comprehensive ablation study is performed to verify the efficacy of each component proposed in our framework. Datasets.Our experiments are conducted using the MVTec-AD [2] and VisA [52] datasets. MVTec-AD is composed of 15 subsets and encompasses a diverse range of defect types, such as scratches, dents, and contaminations, providing comprehensive coverage across various industrial sectors. VisA comprises 12 subsets, covering a range of structural anomalies such as misalignment or missing components, along with other defect types including cracks, corrosion, and more.Evaluation metrics. For classification, we utilize the Area Under the Receiver Operating Characteristic curve (AUROC) and Average Precision (AP) as evaluation metrics. AUROC measures the trade-off between sensitivity and specificity, while AP considers the precision-recall trade-off. For segmentation, in addition to pixlevel AUROC, we also use Per-Region Overlap (PRO) as an essential metric in evaluating segmentation performance. In this paper, we use P-AUR and P-PRO to denote AUROC and PRO at the pixel level metric, respectively. Similarly, I-AUR and I-AP represent AUROC and PRO metrics at the image level.Implementation details. We employ the pre-trained CLIP model developed by OpenAI, where the vision branch is based on the ViT (Vision Transformer) architecture. The utilized hierarchy 𝐿 = {6, 12, 18, 24}. Training is conducted to one NVIDIA-3080Ti GPU over 5 epochs, using the Adam optimizer with a learning rate of 1e -3 to update model parameters. We finetune SimCLIP on the MVTec-AD dataset and evaluate its generalization performance on Visa. Similarly, we finetune SimCLIP on the VisA dataset and evaluate its generalization performance on MVTec-AD.We assess the performance of SimCLIP for zero-shot anomaly detection tasks using two benchmark datasets and conduct a comparative analysis with CLIP-AC [30] , APRIL-GAN [6] and AnomalyCLIP [51] .The result of CLIP-AC in anomaly segmentation is poor because the original CLIP only focuses on high-level global semantic information extracted from images and text during the pretraining phase. APRIL-GAN has achieved promising performance by integrating prompt ensemble and features in its approach. AnomalyCLIP accomplishes better performance by learning object-agnostic prompts. On the VisA, SimCLIP demonstrates superior efficacy, surpassing the second-ranked AnomalyCLIP by 1%/0.6% in AUROC/AP for anomaly classification. On the MVTec-AD, while SimCLIP exhibits inferior anomaly classification compared to AnomalyCLIP, there is an improvement of 0.7% in AUROC and 5.4% in PRO for anomaly segmentation. SimCLIP focuses on realignment between vision and language, enabling the capture of subtle differences in images, and contributing to its better performance in anomaly segmentation tasks.We conducted comprehensive comparisons and analyses with methods including PatchCore [33] , AnomalyGPT [13] , RegAD [16] , APRIL-GAN on two benchmark datasets for both few-shot anomaly segmentation and anomaly classification. PatchCore and RegAD, which concentrate exclusively on vision detection without incorporating multi-modal information, demonstrate performance constraints that impact their competitiveness in few-shot anomaly detection. APRIL-GAN achieves better performance by manually crafting a large set of text prompts and ensemble them to guide V-L models.AnomalyGPT achieves solid outcomes in anomaly segmentation by effectively leveraging a prompt learner to fine-tune large visionlanguage models. SimCLIP+ integrates the relational information among vision features and merges the cross-modal synergy information between vision and language, achieving the best result. In 1-/2-/4-shot anomaly classification, SimCLIP+ exhibits stronger performance when compared to AnomalyGPT, showing enhancements of 5.6%/5.1%/3.8% in AUROC respectively on the Visa benchmark. tion based on an image, implicit prompt tuning offers the following two advantages. Firstly, unlike methods that optimize at the input side of the language branch, implicit prompt tuning optimizes the text prompt at the output side. Not only does IPT avoid the biases introduced by the language branch, which might overly focus on foreground objects in the image rather than anomalies, but it also improves optimization speed by not requiring the gradient to backpropagate through the entire language branch. Secondly, implicit prompt tuning effectively complements domain knowledge by introducing task-specific prompt embeddings for anomaly detection tasks, enhancing the model's context understanding and discriminative capability.In addition to comparing the performance between IPT and SOTA prompt learning methods, we also assessed their differences in computational and memory overhead. As shown in Table 2 , With a similar number of parameters to other methods, IPT requires fewer Flops, resulting in higher computational efficiency.Multi-Hierarchy Vision Adapter and Implicit Prompt Tuning.We focus on investigating the effectiveness of two critical modules on the generalization performance of pre-trained CLIP. From this perspective, we start by removing the MHVA and IPT modules, optimizing only the projection of the vision and language branches during fine-tuning, which serves as our baseline. Subsequently, the MHVA and IPT modules are integrated by us separately into the baseline to compare unidirectional with bidirectional adaptation.Based on the baseline results as shown in Table 3 , it is evident that the original CLIP is more adept at handling AC tasks. However, the performance in downstream AS tasks is somewhat underwhelming, further highlighting the gap between CLIP and downstream AD tasks. Although unimodal adaptation of either MHVA or IPT to some extent alleviates this issue, the potential of CLIP has not been fully realized. By employing cross-modal realignment through bidirectional adjustment of both the MHVA and IPT modules, there was an improvement of 72.7% and 15.7% in AUROC for AS and AC tasks compared to the baseline, respectively. Adapter Ensemble for Multi-Hierarchy.In this study, we investigate the effectiveness of ensemble visual adapters across various hierarchies. We conduct a set of control experiments comprising two different conditions: 1)Using a multi-hierarchy vision adapter same as SimCLIP. 2)Using only a single-hierarchy vision adapter where the hierarchy is a subset of the multi-hierarchy level. Additionally, pixel-level AUROC and PRO are used as performance evaluation metrics in our study. As shown in Table 5 , the multi-hierarchy vision adapter outperforms the other four single-hierarchy vision adapters, demonstrating its effectiveness. Compared to the second-best performing single-hierarchy vision adapter, the multi-hierarchy vision adapter shows improvements of 0.9% and 3.9% in pixel-level AUROC/PRO, respectively.Feature-driven method and prior-aware optimization algorithm.Furthermore, we explore the effectiveness of the featuredriven method and the prior-aware optimization algorithms within SimCLIP+ in the few-shot settings. In this study, We consider three scenarios: 1)Using single feature-driven methods. 2)Using single prior-aware optimization algorithms. 3)Integrating feature-driven methods with prior-aware optimization algorithms at the same time. Table 6 reports the ablation result, using either a feature-driven method or a prior-aware optimization algorithm alone showing promising results in anomaly segmentation tasks. However, for anomaly classification tasks, these single methods often struggle to achieve accurate classifications. Combining feature-driven methods with prior-aware optimization algorithms leverages both vision embedding and cross-modal information, significantly enhancing anomaly segmentation and anomaly classification performance compared to using either method alone in the 1-/2-/4-shot settings.Figure 4 reports the visualization results of SimCLIP/SimCLIP+ on two benchmark datasets. Through realignment of vision and language, SimCLIP can accurately locate anomalies in images, including the ability to detect multiple anomalies within a single image(e.g., the 'Wood' in Figure 4 ). The qualitative analysis demonstrates that SimCLIP+ outperforms SimCLIP in anomaly segmentation. This indirectly highlights the effectiveness of SimCLIP+ in integrating both vision embedding information and cross-modal synergy information concurrently.As shown in Figure 5 , We employ t-SNE [40] to visualize the distances between normal prompt and anomalous prompt embedding in the feature space. Figure 5 (a) shows that there are no distinct boundaries separating the different types of text prompts in the feature space. This severely hinders CLIP's application in downstream anomaly detection tasks. The effectiveness of IPT is visually demonstrated by the clear boundaries observed between different types of text prompts in the feature space after refining, as illustrated in Figure 5  
conclusion: 
 In this paper, we propose a novel method, named SimCLIP that employs bidirectional adaptation to accomplish realignment of vision and language, enhancing CLIP's zero-shot generalization performance on downstream AD tasks. In the vision branch, we incorporate a multi-hierarchy vision adapter situated at various levels to capture more intricate spatial details and facilitate efficient cross-modal interactions with language. In the language branch, we employ a learnable task-specific prompt embedding and the implicit prompt tuning module to refine the original prompts. Sim-CLIP bridges the gap between CLIP and downstream zero-shot AD tasks by bidirectionally adjusting both two branches. Additionally, we further propose SimCLIP+, which integrates correlation information among vision embeddings with cross-modal synergy information, coupled with a prior-aware optimization algorithm to address AD tasks under limited normal samples. Our proposed method provides a new perspective on bridging the gap between pre-trained vision-language models and downstream zero/few-shot AD tasks.  
: 
 AD task. Therefore, applying CLIP directly to zero-/few-shot AD tasks inevitably encounters two challenges.On the one hand, the strong generalization of CLIP relies on text prompts aligned to images. In real-world scenarios, the diverse types of anomalies make it challenging for general text prompts to encompass them comprehensively. Existing approaches [6, 17] mitigate this problem by integrating expert knowledge and manually crafting extensive prompts (Figure1(a)). However, its effectiveness is the necessity for verification with a certain degree of prior knowledge. Besides, creating substantial hand-crafted task-specific prompts for each scenario is time-consuming. Recent prompt learning works (e.g., CoOp [49] ) can address this by directly learning prompts from training data of downstream tasks. Such methods can obtain better prompts in contrast to hand-crafted ones, but the learned prompts are bounded by the distribution associated with training data and have limited generalization [35] . In addition, the learned prompts through this approach lack interpretability and semantic coherence, which contradicts the original intention of CLIP aligning image-text pairs. This inspires us to explore an approach that can effectively guide the CLIP in accomplishing AD tasks using only simple prompts.On the other hand, CLIP is an image-text aligning Pre-trained model well-suited for classification tasks. Nonetheless, AD encompasses the task of Anomaly Segmentation (AS), demanding pixellevel localization of anomalies. To accomplish AS task, current methods [6] use linear mapping to transform patch-level features into a multi-modal feature space. Subsequently, AS is achieved by calculating the similarity of features between image patches and text prompts in this embedding space (Figure 1(a) ). Due to the absence of changes in the prompt features, this process entails adapting fine-level patch features to high-level semantic prompt features. However, this unidirectional adaptation is insufficient to fully exploit the potential of CLIP in AS tasks. This insight motivates an exploration into the question: How to effectively align patch features with text features in the multi-modal feature space?In this paper, we propose a novel approach, named SimCLIP, designed to bridge the gap between CLIP and downstream AD tasks in zero-/few-shot scenarios. As shown in Figure 1 (b), SimCLIP accomplishes realignment of vision and language through bidirectional interaction adjustments of both two branches. For the vision branch, SimCLIP employs a multi-hierarchy vision adapter (MHVA) at different hierarchies to capture richer spatial information and enable effective cross-modal interactions with language. In the language branch, we leverage task-specific prompt embedding to learn relevant knowledge for AD tasks, alongside employing implicit prompt learning (IPT) to refine and optimize the original prompt. SimCLIP presents the following advantages: 1) Cross-modal realignment significantly enhances the performance of the pre-trained CLIP model in downstream AD tasks, demonstrating superior zero-shot generalization compared to the initial CLIP, as shown in Figure 2 . 2) By employing prompt learning within a latent embedding space to refine initial prompts, we overcome the limitations imposed by text information capacity and broaden the solution space. 3) Implicit prompt learning significantly reduces the dependency on numerous hand-crafted prompts typically required for V-L models, enhancing the flexibility and efficiency of the model. 4) SimCLIP maintains the interpretability and semantic coherence of the text by fusion simple prompts with task-specific prompts in embedding space. Furthermore, we introduce SimCLIP+, an extended version of SimCLIP, integrating the relational information among vision features and merging the cross-modal synergy information between vision and language to address few-shot AD tasks.In summary, we make the following main contributions:• We propose a novel zero-shot AD approach named SimCLIP, which achieves realignment on pre-trained CLIP through bidirectional adjustment of both the multi-hierarchy vision adapter (MHVA) and implicit prompt learning (IPT). This method simplifies prompt design and seamlessly adapts to anomaly detection tasks involving unseen classes. • To tackle AD tasks under limited samples, we introduce an extended version of SimCLIP, called SimCLIP+. This method not only utilizes intrinsic correlation information among vision embeddings but also integrates the cross-modal synergy information generated by SimCLIP. To further enhance anomaly detection efficiency, we also propose a prior-aware optimization algorithm designed to optimize the cross-modal synergy information mentioned above. • We conduct extensive experiments on MVTec-AD [2] and VisA [52] benchmarks. SimCLIP/SimCLIP+ achieves superior performance for zero-/few-shot anomaly classification and segmentation compared to the state-of-the-art. 
 
 Appendix:  


Decision: This paper has been evaluated by 4 reviewers who all converge to the Borderline accept score. This score results from the merits of the paper and from the high quality elements found in the rebuttal, which helped reviewers to get a better understanding of some of the elements highlighted in the weaknesses sections of their evaluations. Authors make promises in the rebuttal -- that has to be implemented in the final version of the manuscript in order to improve the overall quality of the article, thanks to the issues highlighted by the reviewers.

All these reasons push the recommendation to be ACCEPT, poster.

Reviews: review: paper_summary: paper_summary: The manuscript presents a novel approach named SimCLIP, aimed at refining the misalignment between high-level language features and fine-level vision features in anomaly detection (AD) tasks. This is achieved through the bidirectional adjustment of both a Multi-Hierarchy Vision Adapter (MHVA) and Implicit Prompt Tuning (IPT). Additionally, an extended version, SimCLIP+, is introduced for few-shot scenarios, incorporating relational information among vision embeddings and cross-modal synergy. The results from extensive experiments on two challenging datasets, MVTec-AD and VisA, indicate superior performance in zero-/few-shot anomaly classification and segmentation compared to current methods.
strength_weakness: strength_weakness: 1. The paper successfully integrates multiple advanced techniques, including MHVA and IPT, to address the alignment issues between vision and language modalities, which is a significant contribution to the field of anomaly detection.
2. The method simplifies the process of designing text prompts for AD tasks, potentially reducing the dependence on expert knowledge and making the approach more accessible.
3. The experiments results support for the effectiveness of the proposed approaches, SimCLIP and SimCLIP+.
limitations: limitations: 1. This paper does not clarify whether there is a comparison of training costs and efficiency between SimCLIP/SimCLIP+ and other models, which is essential for evaluating practical deployment feasibility.

2. Parameter control is not specified if the comparisons of individual components in Table 3 control for total parameter count or computational cost, potentially affecting the accuracy of the comparative analysis.

3. There is a lack of performance data on more challenging datasets, which is necessary to fully assess the robustness and generalization capability of the proposed methods.

4.  The manuscript could be strengthened by more clearly highlighting distinctive innovations or unique aspects of SimCLIP and SimCLIP+ that distinguish them from existing methodologies.
paper_summary: paper_summary: The author proposes a method called SimCLIP, which improves the inconsistency between high-level language features and fine visual features through bidirectional adaptation of multi-level visual adapters and implicitly prompted tuning.
strength_weakness: strength_weakness: 1. The paper has a clear motivation, the method is consistent with the motivation, and the paper is presented clearly.
2. The proposed method has achieved performance improvement on multiple datasets.
limitations: limitations: 1. The model is trained using both the training set and another dataset. Will it cause leakage of new classes? I believe that ablation experiments should be added to obtain the performance results of models that do not use data from the other dataset.
2. Whether the methods compared in the paper also used additional datasets, and whether the backbone is consistent with this paper, I believe that these two points should be taken into consideration to ensure fairness. The authors can also try the results of other backbones.
paper_summary: paper_summary: To deal with the misalignment of high-level language features with fine-level vision features in anomaly segmentation tasks for large pre-trained vision-language models, this paper proposes the SimCLIP to refine the misalignment. Specifically, SimCLIP employs a multi-hierarchy vision adapter at different hierarchies for the vision branch. For the language branch, SimCLIP employs implicit prompt learning to refine and optimize the original prompt. Besides, this paper introduces an extended version of SimCLIP, named SimCLIP+, to utilize intrinsic correlation information among vision embeddings and integrate the cross-modal synergy information generated by SimCLIP. Finally, extensive experiment results on MVTec-AD and VisA benchmarks demonstrate the robustness of the proposed SimCLIP and SimCLIP+.
strength_weakness: strength_weakness: 1.	This paper is well-composed with a clear organizational structure. Figures 1, 2, and 3 effectively illustrate the motivation and detailed content of the paper, aiding in the comprehension of the author’s concepts and improving the overall readability for the audience.
2.	The paper provides the detailed algorithm pseudocode (i.e., as shown in Algorithm 1) and detailed code in the github link, demonstrating the reliability and reproducibility of the results.
3.	The experimental section provides qualitative and quantitative analyses, which intuitively and effectively demonstrate the performance of MHVA and IPT proposed for images and text branches, respectively, and validate the robustness of SimCLIP+.
limitations: limitations: 1.	The authors mention in the paper that they design the simple binary prompts to refine the language modality. However, the paper lacks a deeper theoretical analysis and proof of this approach. Providing such analysis would enhance the credibility of the paper. 
2.	The paper lacks ablation analysis and experimental results for the hyperparameters (e.g., \tau), and specific values are not provided in the experimental details section. The authors need to further supplement the corresponding experimental results.
3.	The experimental comparisons in the paper are insufficient. The authors only compare a few SOTA methods in Tables 1, 2, and 4, which is not enough to prove the superiority of the proposed method. The authors should provide more experimental comparison results with SOTA methods published in the last two years.
4.	The authors should provide specific indices corresponding to the specific parts of the figure in the caption of Figure 3 to facilitate readers' intuitive understanding of the proposed SimCLIP.
paper_summary: paper_summary: The paper proposed a SimCLIP that employs bidirectional adaptation to accomplish realignment of vision and language, enhancing CLIP’s zero-shot generalization performance on downstream AD tasks. The experiments on two challenging datasets verified the effectiveness of the proposed method.
strength_weakness: strength_weakness: (1) This paper explores the zero-shot generalization performance of enhanced CLIP on downstream AD tasks. The proposed method is a promising approach.
(2)The presented method is technically sound. Its key components are clearly explained with clear formulations.
(3) The experiments conducted on the MVTec-AD and VisA datasets demonstrate the effectiveness of the proposed method.
limitations: limitations: （1）The author should carefully proofread the paper as there are several issues with details. 1) There are inconsistencies in descriptions, such as 'Datasets.Our' at line 557; 2) Table 1 is not cited in the text; 3) Some paragraphs are too long, so it is suggested to divide them without changing the content.
（2）The lack of introduction and analysis on recent relevant work [1-2] is noticeable.
[1] Li Y, Goodge A, Liu F, et al. PromptAD: Zero-Shot Anomaly Detection Using Text Prompts[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 1093-1102.
[2] Gu Z, Zhu B, Zhu G, et al. Anomalygpt: Detecting industrial anomalies using large vision-language models[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(3): 1932-1940.
（3）Sections 4.1 and 4.2 describe some results without explicitly indicating which corresponding table of experimental data supports these outcomes.
（4）Evaluation indicator for the experiment is insufficient.
（5）The authors do not clearly articulate the advantages of the proposed method compared to existing methods.
（6）Experiment lacks comparison of computational complexity and inference time.