Language-Assisted Feature Transformation for Anomaly Detection
Authors: EungGu Yun
Heonjin Ha
Yeongwoo Nam
Bryan Dongik Lee
Venue: ICLR 2025 Poster (2025)
Keywords: anomaly detection
feature transformation
vision-language model
language guidance
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present Language-Assisted Feature Transformation (LAFT) to guide normality boundaries in image anomaly detection using natural language.

Abstract: This paper introduces LAFT, a novel feature transformation method designed to incorporate user knowledge and preferences into anomaly detection using natural language. Accurately modeling the boundary of normality is crucial for distinguishing abnormal data, but this is often challenging due to limited data or the presence of nuisance attributes. While unsupervised methods that rely solely on data without user guidance are common, they may fail to detect anomalies of specific interest. To address this limitation, we propose Language-Assisted Feature Transformation (LAFT), which leverages the shared image-text embedding space of vision-language models to transform visual features according to user-defined requirements. Combined with anomaly detection methods, LAFT effectively aligns visual features with user preferences, allowing anomalies of interest to be detected. Extensive experiments on both toy and real-world datasets validate the effectiveness of our method.

Published as a conference paper at ICLR 2025
LANGUAGE-ASSISTER FEATURE TRANSFORMATION FOR ANOMALY DETECTION
EungGu Yun SAIGE
Heonjin Ha! LG Uplus Scoul, South Korea
Yeongwoo Nam Alsemy Inc. Scoul, South Korea
Bryan Dongik Leef Independent Seoul, South Korea
Scoul, South Korea
ABSTRACT
This paper introduces LAFT, novel feature transformation method designed incorporate user knowledge and preferences into anomaly detection using natu- ral language. Accurately modeling the boundary normality crucial for dis- tinguishing abnormal data, but this is often challenging due to limited data or the presence nuisance attributes. While unsupervised methods that rely solely data without user guidance common, they may fail to detect anomalies specific interest. To address this limitation, we propose Language-Assistee Feature Transformation (LAFT), which leverages the shared image-text embed- ding space of vision-language models to transform visual features according to user-defined requirements. Combined with anomaly detection methods, LAFT effectively aligns visual features with user preferences, allowing anomalies of in- terest be detected. Extensive experiments on both toy and real-world datasets validate the effectiveness of our method.
INTRODUCTION
Anomaly detection (AD) is the task of distinguishing abnormal data that deviates from the norm. In most scenarios where anomaly detection is applied, normal data is relatively easy to obtain, while abnormal data is scarce sometimes impossible obtain in advance. Thus, typical anomaly de- tection methods rely on normal data provided by users to learn what constitutes normal. However, when the training data is biased or does not cover the diverse variations, modeling the boundary of normality becomes a significant challenge (Lee & Wang, 2020; Cohen et al., 2023). In practical applications, models may need prioritize or disregard certain attributes the data. For instance, when inspecting products in images, a user might focus solely on the product's shape, ignoring attributes like color or lighting conditions. Moreover, distinguishing anomalies becomes more diffi- cult when attributes are entangled, seen in the Waterbirds dataset, where the background and bird features are entangled (Sagawa et al., 2019).
To address this issue, various methods have been proposed that use data augmentation generation techniques to improve the learning of decision boundaries (Zavrtanik et al., 2021; Li et al., 2021; Du et al., 2021). These approaches aim produce more diverse samples, covering a broader range of the underlying data distribution than what is available. Additionally, some approaches focus on enabling models learn task-specific feature representations (Chen al., 2020a;b; Caron et al., 2020), applying them to anomaly detection to better capture feature-level normality (Hyun et al., 2023). However, limitation of these methods is that they may struggle generalize to completely unseen data or fail to align with the user's intent in defining normality.
In some scenarios, users may have prior knowledge or specific preferences about the data that they want to integrate into the anomaly detection process. Typically, this is achieved through indirect methods, such as manually applying random color augmentation to ignore certain object colors. Controlling the boundary of normality remains relatively unexplored, and existing approaches often require unrealistic conditions, such as access to anomaly samples or labels (Cohen et al., 2023). To overcome this limitation, we propose leveraging vasion-language models to directly integrate user knowledge and preferences into the anomaly detection framework through natural language. By
Conresponding author: eg yun8 saaigeeai ai This work was primarily conducted while the authors were SAIGE.

---

Published as a conference paper at ICLR 2025
Criteria: shape
Train
Test
Train
Test
Typical
LAFT AD
AD
Normal
Anomaly
Normal
Anomaly
Normal
Anomaly
Normal Anomaly
High-level motivation of our method: (left) typical image anomaly detection methods treat all test data that differs from the training data anomalies, while (right) method, LAFT AD, incorporates user preferences into the anomaly detection.
Figure 1:
using language, users more explicitly express their desired concepts, providing greater control over the definition of normality.
Recent studies have shown the effectiveness of training vision models with large amounts of unla- beled Internet data (Radford et al., 2021; Jia et al., 2021: Desai et al., 2023). By using image-text pairs from the web for pre-training, these models natural language descriptions quality of image representations. Through large-scale training, they can correlate visual concepts in images with textual descriptions, aligning image and text features in shared embedding space. Researchers have applied these models to industrial anomaly detection (Jeong et al., 2023; Cao al., 2023b; Chen et al., 2023; Zhu & Pang, 2024) and general image out-of-distribttion tasks us- ing zero-shot text prompts (Ming et al., 2022: Miyai et al., 2023). A key benefit is the ability to incorporate human knowledge through text prompts, allowing zero-shot use without requiring addi- tional training images. However, defining the complex normality of an image solely through natural language remains a challenge, and many methods face structural limitations in utilizing available training images. Therefore, image features, similar to other image anomaly detection approaches, with language serving only to refine the boundaries of normality.
improve the
aim develop method where normality is primarily defined by
In this paper, we present Language-Assinn Assisted Feature Transformation (LAFT), a method that allows users to control the transformation of image features using natural language without requiring ad- ditional training. LAFT leverages the vision-language model CLIP (Radford et al., 2021), using its shared embedding space link visual and textual features. This connection enables the trans- formation of visual features based solely on language inputs. We hypothesize that visual concept subspaces exist within this shared embedding space, and introduce the notion of "concept axis" to represent these subspaces. By computing pairwise differences between textual features, we derive concept difference vectors that define the concept axes. Projecting visual features onto orthogonal to these axes allows for selective emphasis or suppression of specific image attributes.
Our method offers training-free free approach by using language, whereas most feature transformation methods require extensive training on data. This property presents two key advantages: it is agnostic to downstream tasks and performs well even in settings where data is scarce. By combining LAFT with proper anomaly detection methods, we can apply LAFT to anomaly detection tasks. Our ap- proach differs from most work using CLIP for anomaly detection in that relies primarily on image features to define normality, with language playing supporting role. By using language, users can provide their understanding of normality, allowing greater flexibility in incorporating domain knowledge. Furthermore, by defining the boundaries of normality using image features, the model can accurately distinguish between normal and abnormal images.
We summarize contributions follows:
1. We propose Language-Assisen Feature Transformation (LAFT), novel method that uses natural language transform image features fit the given task requirements by leveraging the image- text aligned embedding space of CLIP.
2. We introduce LAFT AD, an anomaly detection method that combines LAFT with k-nearest neighbor (kNN) classifier, enabling users selectively focus on or ignore specific image at- tributes based on their guidance for semantic anomaly detection tasks.
3. We present WinCLIP+LAFT,
extension of WinCLIP that integrates LAFT to improve perfor- mance in industrial anomaly detection tasks.
We demonstrate the effectiveness of our method on Colored MNIST and extensively evaluate its performance on real-world datasets, including Waterbirds, CelebA, MVTec AD, and VisA.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Image anomaly detection with vision-language model Since the advent of CLIP (Radford et al., 2021), numerous studies in image anomaly detection have attempted to exploit the generalization capability of this lasion-language model. To align visual and textual features for effective out-of- distribution detection, Ming et al. (2022) proposed a scoring method called MCM, with Miyai et al. (2023) later presenting an improved version, GL-MCM. Addressing the limitations of zero-shot, Ming & Li (2023) tried to further enhance performance by using parameter-efficien fine-tuning in downstream tasks. Fort et al. (2021) aimed to improve the model's understanding normality by providing the CLIP text encoder with candidate anomaly labels. In addition, Esmaeilpour et al. (2022) introduced framework for training label generator based on the CLIP image encoder to generate possible anomaly labels. For industrial anomaly detection, Jeong et al. (2023) introduced WinCLIP, a zero-/few-shot anomaly detection model that efficiently extracts and aggregates features at multiple levels, aligning them with textual information. Similarly, Chen et al. (2023) proposed APRIL-GAN, and Zhu & Pang (2024) developed InCTRL,, both of which adapt CLIPP image features using additional adapter layers to better align them for anomaly detection, although these approaches require additional pre-training of the adapter layers.
Adjusting the normality boundary Few studies have specifically addressed the challenge of ad- justing the normality boundary anomaly detection. Cohen et al. (2023) introduced Red PANDA, an anomaly detection method that disentangles relevant attributes in images while ignoring nuisance factors. However, achieving this disentangled feature representation requires labeled data for each nuisance attribute. Reiss et al. (2023) emphasized that overly expressive feature representations can ultimately degrade performance, highlighting a trade-off between sufficient representation and ever-expressiveess: in anomaly detection. Hendrycks et al. (2018) introduced outlier exposure, an approach that uses auxiliary data to help models generalize more effectively to unseen anomalies.
Extracting task-specific features Several strategies have been developed to enhance the adapt- ability and robustness of features extracted from backbone models. Some studies focus on fine- tuning pre-trained feature extraction backbones or generating task-specific features through feature transformations. Ruff et al. (2018) introduced a method that transforms normal data into a hyper- sphere representation for anomaly detection, and Reiss et al. (2021) proposed an early stopping strategy to prevent feature collapse. Chen et al. (2020a:b) utilized contrastive pre-training to facili- tate feature agreement, and Caron et al. (2020) employed prototype vectors for contrastive training of similar features. Following this line of research, Hyun et al. (2023); Reiss & Hoshen (2023); Tack al. (2020) extended contrastive learning approaches for anomaly detection. Zhao al. (2023) suggested using the backbone of vision-language pre-trained diffusion models and training a text adaptor to extract spec-fc features with text prompts for downstream tasks.
3 PRELIMINARIES
In our scenario, training set, represented Dirain. consists of normal samples only, and a test set Diest consists of both normal and anomalous samples. For a two-stage anomaly detection model consisting of feature extractor and an anomaly classifier g. the feature extractor f maps the input image x to a feature v = f(x), and the anomaly classifier g maps the feature v to an anomaly score (v). Then, the anomaly score s; is used determine the prediction of the anomaly label ÿ..
The attributes of an image x extracted by the feature extractor are denoted as = {aa ... a""), and the anomaly label is denoted as y. Each attribute a² (j 1, 1,... m) denotes any characteristics within the feature extracted from the image, such as the shape of the object, the color, or the back- ground. The m attributes can be divided into relevant attributes n {al Jisisn and irrelevant (nuisance) attributes ah = {al Jnejsm for desired anomaly detection tasks. For example, when detecting anomalies in the shape of objects, the shape is relevant, while the color is irrelevant. To properly detect anomalies, the prediction of the model should be invariant to the irrelevant attributes.
There are two ways to achieve this invariance:
1. Provide enough data that covers the possible values for each a², so that the classifier g can properly ignore irrelevant attributes "" in the feature v. This is the most desirable solution, and many data augmentation and generation methods have been proposed. However, it is often impossible collect or hard to generate such data.
3

---

Published as a conference paper at ICLR 2025
LAFT
Concept values
minnale aguare
Cunaeyt proctypes
PCA
plaso of
difference
Concept subapsoe
images
G Iroge Encoder
Train image features
(Trersfomnal) Trzin image feanures
Projection
Classifer
images
Anomaly
CLIP Image
Test inagee
Projection
(Irasfoomed)
Test
image
Overview of our method, LAFT, transformation module, and LAFT AD, combining LAFT with kNN classifier, Our approach uses CLIP's text and image encoders without any addi- tional training. The key idea is to use text prompts containing concept values construct a concept subspace for the target attribute. This process involves computing pairwise differences of concept prototypes and extracting robust concept axes via PCA. Once the concept subspaces are created, the shared embedding space can used to transform image features suitable for anomaly detection.
Figure 2:
Make the feature extractor only extract the relevant attributes ael and do not include the irrel- evant attributes ! in the feature v. Fine-tuning the feature extractor or adding transformation to the feature extractor common way to achieve this. But is often hard to design such training procedures to achieve the invariance.
Our goal is to design a transformation that transforms the feature e into a new feature d' = Twv) that the anomaly classifier g can to detect anomalies without being affected by the irrelevant attributes a using only the user's natural language without any additional training data or labels.
This can be achieved by two approaches:
Guide Make transformation T Tguide that includes only the relevant attributes a² a'd Some attributes in rd may be correlated, so the transformed feature may not include all relevant attributes.
Ignore
Make transformation Tignore that excludes all irrelevant attributes Val a In many cases this is harder to achieve than the above approach, because the transformation should be able to remove all irrelevant attributes.
That we want our transformation T represent the relevant attributes in a manner unaffected by the irrelevant attributes:
p(a" n+1
a) p(a"
T()))-
We also want the transformed feature o' to be informative, containing enough information about relevant attributes. Here, I(:) represents the mutual information between the two arguments:
T((a aa"); v) T((a), ((a')) T())).
In practice, invariance can be measured by the accuracy of predicting the anomaly label á from the transformed feature T(v). But we can assess the informativeness by measuring the accuracy of predicting the relevant attribute utilized define anomalies. Empirical evaluations of these measures for various datasets are provided in the Experiments. With such representation, anomalies later be evaluated independently, devoid of any bias caused by the irrelevant attribute we aim to disregard.
CLIP (Radford et al., 2021) embeds the features in unit sphere subspace in Euclidean space IR" An embedding vector of an image is correlated to the text embedding describing the image. This construct the transform with the CLIP text encoder. We assume that all relevant and irrelevant features can be encoded with the text description, so that natural language assists in the manipulation of the vector in the CLIP shared embedding space.
means that
4

---

Published as a conference paper at ICLR 2025
4 METHOD
Anomaly detection often faces data scarcity, especially for abnormal data, making it diffccutt for models to define the normality boundary. In this situation, users may have prior knowledge or preferences about what should be considered normal. Therefore, we aim to design:
A method that can be used when the user has knowledge of normality and wants control it. Typical anomaly detection methods consider all test data different from the training data as anomalies, but we want our method to be used only with language.
method that effectively handles anomalies that are challenging to express solely in natural language. Other methods using vision-language models require normality to be expressed entirely in language prompts for guidance, which limits the ability to capture complex normality. We want method to utilize image features to define normality.
Note that our method not intended to be used in situations where the user has no knowledge or preferences, or to automatically identify relevant attributes.
Our method uses CLIP's text and image encoders without additional training. The core idea is to construct a concept subspace for target attributes using text prompts containing concept values, Af- ter constructing the subspace, transforms image features by projecting them onto the subspace, taking advantage of CLIP's shared embedding space. This involves computing pairwise differences between textual features containing concept prototypes to extract robust concept axes. In this sec- tion, we provide a detailed explanation of our method, illustrated in Figure 2.
4.1 TEXT PROMPT
To guide the model in focusing on or ignoring specific attributes of an image, it is essential to provide it with appropriate textual prompts. Following Ming et al. (2022), we assume that the text contains concept prototypes representing the attributes. Thus, we provide the method with a list of prompts composed of templates and values, as commonly done CLIP-bbased methods (Radford et al., 2021; Ming al., 2022; Jeong et al., 2023). The key difference in approach is that we use the actual values of the desired attribute (c.g., "circle," " "square") rather than the attribute name itself (c.g., "shape"). For instance, to capture the concept of hair color, construct the prompt
"a photo of a person with brown huir"
"a potrait of man with black hair"
"an image a blond child"
By using the actual values of the desired attribute in the prompts, we aim for the method to capture the difference between the concept prototypes of the attribute. Providing values for this attribute that are not present in the training set, but are likely to appear during testing, helps construct a more comprehensive subspace for that concept. As with other language-based methods, multiple types of templates can be provided to mitigate the bias introduced by any single template. We examine the effect of various sets of concept values in the Ablation Study, with the prompts used in our experiments detailed in the Appendix.
4.2 FIND CONCEPT SUBSPACE
Mikolov (2013) showed that simple arithmetic operations between text embeddings can capture meaningful relationships (e.g., vec(biggest) vec(big) AS vec(sma limaaleess llest) vec(smal (1)) This finding showed that text embeddings not only represent texts in a vector space, but also encode the underlying relationships between them. Moreover, they observed that dighedimensional vectors, when trained on large datasets, are capable of capturing subtle semantic relationships. Similarly, CLIP's text embeddings support arithmetic operations to compute differences between concept pro- totypes, allowing for the comparison of these concepts in the emhedding space.
Building this approach, the method construets a subspace of the concept within CLIP's embed- ding space. Specifically, it identifies the axes of this subspace that capture the variance between concept prototypes, as represented by difference between the prompts. For prompts 4; and Lj. where < i < j < n. we compute the pairwise differences of the text features:
! Uij = Frcu (1:) (1z)
(3)
5

---

Published as a conference paper at ICLR 2025
features
transformed
Attributt Si ree
Axtre 2
Color
5 /grzen
Axis
Number
Figure 3: Projection image features from CLIP's image encoder (left) and transformed image features using LAFT (right). Without guidance, the image features may not align with the intended attributes. After applying LAFT, the features become more aligned with the desired attributes.
where n represents the number of prompts, and E denotes CLIP's text encoder. But directly using these vectors is not preferable because the prompts may contain template noise (Zhou et al., 2022). To address this, we apply PCA to extract the principal axes from these vectors:
IsAss = PCA ((A)) hsssssn.)) d)
(4)
where d is the number of components, and {cx} represents the principal axes, collectively re- ferred to as the concept axes. Throughout this paper, we typically select d between 8 and 32 when guiding an attribute, and between 32 and 384 when ignoring an attribute. As discussed in the Pre- liminaries, ignoring attributes is generally more challenging than guiding them, so larger number of components is used. For the impact of d, please refer to the Ablation Study.
4.3 FEATURE TRANSFORMATION WITH PROJECTION
For each image feature u; = f(x)) encoded by CLIP's image encoder, we project the features onto the concept subspace:
of = guide (1:) = Σ (ui, (x)
(Cks Ck) Ck+
where - denotes the inner product. This projection retains only the relevant attributes of the image feature, as irrelevant attributes are nearly orthogonal to the concept axes. Conversely, we can remove the irrelevant attributes using orthogonal projection. Let č represent the concept axes associated with the irrelevant attributes. Then we can project orthogonally onto the concept subspace as follows:
(1i, Ck) (kk, Ck)
T iggmore (w:) = U2
Ck:
which manually cancels out the vectors of irrelevant attributes. This completes the description of the feature transformation method, L.AFT.
4.4 ANOMALY SCORING
Many anomaly detection methods employ k-nearest-neiighbrrs (kNN) for anomaly scoring (Cohen & Hoshen, 2020; Roth et al., 2022). This approach is effective because normal data tends to be densely concentrated, while anomalous data is typically sparsely distributed in the feature space. In our method, LAFT AD, we also use kNN to estimate the density of normal data around each test sample, assuming that the features have been processed by LAFT for semantic anomaly detection.
We start by extracting features for each normal sample: vf (x:())), Vz; E D Next, for each test sample, we infer its feature: v/ = T(f(x,)), Vn, Dress- Finally, we score each test sample
based on its ANN distance from the normal data:
8j ((()))
1
Σ
Sana (1 v/)
T
()NN((
where (x!) denotes the k nearest features to v' in the normal data, and Scos (:.)) represents cosine similarity. We use 30 for &NN throughout the paper without optimization.
However, this method may not be suitable for industrial anomaly detection tasks, where anomalies are often small and subtle. To address this, we extend WinCLIP (Jeong et al., 2023) to incorporate LAFT for anomaly scoring, which we refer to as WinCLIP+LAFT for industrial anomaly detection. detailed explanation of this extension is provided in the Experiment.

---

Published as a conference paper at ICLR 2025
EXPERIMENTS
Datasets To validate our approach, we used the colored version of MNIST (LeCun et al., 2010), Waterbirds (Sagawa et al., 2019), and CelebA (Liu et al., 2015) datasets for semantic anomaly detection (SAD). We defined normal and anomalous values for each dataset attribute and divided the training split into 2 subsets. For example, in the Colored MNIST dataset, we designated digits 0-4 as normal and 5-9 as anomalous, with the color red as normal and green and blue as anomalous. We then used one subset as the training set, considering normal across all m attributes (e-g., digits 0-4 and the color red). This is similar to the setup commonly used in many studies for a single attribute (primarily based on class labels) (Ruff' et al., 2020; Tack al., 2020; Esmaeilpour et al., 2022; Cohen et al., 2023; Reiss & Hoshen, 2023; Cao et al., 2023a; Zhu & Pang, 2024). To further demonstrate the practicality of our method, we also used the MVTec AD (Bergmann et al., 2019) and VisA (Zou et al., 2022) datasets for industrial anomaly detection (IAD).
Baselines For semantic anomaly datasets, we do not compare with typical image-only AD meth- ods for two reasons: (1) they require attribute-ppecific processing (e.g., color augmentation) to in- corporate user prior knowledge, which limits generalizability to other contexts, and (2) without guidance, these methods detect images that differ from the training set, resulting in high false pos- itive rates in our settings. Instead, to simulate image-only AD methods, we compare with simple kNN and LinearProbe with additional training data, directly using image features from CLIP.
kNN computes the distance between the test image features and the training image features for anomaly scoring. As discussed in the Method, many AD methods rely on &NN-based scoring, making it an important baseline. &NN using the same training subset as the other methods serves as no-guidance version of LAFT AD, allowing to directly evaluate LAFT's effectiveness. And kNN using additional normal training subset depending on the target attribute represents an image- only method with attribute-spccifie image processing. Since applying image augmentation across all datasets and attributes is not straightforward (e.g., augmenting the background in Waterbirds), we assume that the additional data is well augmented images for the target attribute. To evaluate CLIP image encoder's performance, we provide full training data including normal and anomalous images for LincarProbe to train a linear classifier to predict the class of the test image (Radford et al., 2021).
We also evaluate CLiP-based zero-shot and few-shot AD methods. For zero-shot AD, we use Maxi- mum Concept Matching (MCM; Ming et al., 2022), which requires only prompts for normal images to perform anomaly scoring, and Zero-shot outlier exposure (ZOE; Fort et al., 2021), which uses prompts for normal images and candidate prompts for anomalous images. And CLIPN (Wang et al., 2023) is zero-shot method that uses pre-trained "no" prompts and "no" text encoder to make text features for anomalies. We also consider WinCLIP (Jeong et al., 2023), which supports both zero- /few-shot AD. The zero-shot version of WinCLIP is similar to ZOE in terms of anomaly scoring, and the few-shot version (WinCLIP+) requires a few normal images. Lastly, we consider InCTRL (Zhu & Pang, 2024), few-shot AD method similar to WinCLIP+.
We use the actual class names from the dataset for concept values, if available, and add other candidate labels to simulate unseen classes. For example, for the number attribute in the Colored MNIST dataset, we use '0' to '20' and 'zero' to 'teetty' as number attributes, even though the dataset only includes 0 to 9. We referenced the prompts provided by CLIP (Radford et al., 2021).
Prompts
For more details, please refer to the Experimental Details.
5.1 SEMANTIC ANOMALY DETECTION
We used the colored version of the MNIST dataset (LeCun et al., 2010), similar to Arjovsky et al. (2019), to demonstrate our concept in the simplest way. We created a dataset that divides each digit of the MNIST and colors each split with red, green, and blue. In this way, the image of a colored MNIST consists of two attributes: number and color. We mark the numbers 0 to 4 as normal and the numbers 5 to 9 as abnormal. In addition, we label red as normal and green and blue as abnormal colors. In this setting, the training set consists of 0 to 4 and red images. Then, we use five different seeds to split the training set for coloring each digit. Figure 3 shows brief overview of our desired transformation using concept axes. we choose an axis (number or color) to project the image features, we can simply use ANN to detect only the desired anomalies.
7

---

Published as a conference paper at ICLR 2025
Table 1: Anomaly detection performance (%) on Colored MNIST and Waterbirds datasets. Standard deviations computed over five different seeds, with results for deterministic cases omitted. The best values are shown in bold, and the second- best values are underlined.
Colored MNIST: Number
Waterbirds: Bird
Guidance
Method
AUROC
AUPRC
FPR95
AUROC
AUPRC
FPR95 +
Baseline
Subset of normal images
KNN
92.4 0.2
91.8
31,9 +.9
82.3
91.2
48.1
All normal images
KNN
98.0 =
97.1 0.0
7.5 0.2
83.0
91.5
44.7
Anomalous images
LinearProbe
99.8 0.0
99.8 0.0
0.5 0.1
91.0 0.0
96.7 0.0
34.2 +
Guide
Language
MCM
62.9
52.5
60.8
88.8
95.4
40.0
ZOE
91.2
92.4
47.3
92.2
97.1
32,8
CLIPN-C
73.0 2.5
61.7 3.0
51.0
71.2 + 2.8
1.1
100.0 + 0.0
CLIPN-A
#2
61.6 2.7
50.0 + 0.6
82.3 + 0.8
91.9 0.3
55.6 1: 1.2
WinCLIP
91.1
92.4
48.0
92.2
97.0
32,6
Image+ Language
WinCLIP+
92.6 + 1.3
91.3 2.0
38.8 + 1.5
91.8 + 0.2
96.9 + 0.1
33.4 + 1.5
InCTRL
94.0 + 1.3
92.4 2.6
25.5 4.1.1
83.6 + 1.0
92.0 + 0.7
63.5 #: 3.1
LAFT AD (Ours)
98.5 + 0.0
98.4 + 0.0
6.9 1 0.1
95.6
98.4
20.6
Ignore
Image Language
LAFT AD (Ours)
97.4
96.9 0.2
10.4
10.4444
84.8
92.2
38.6
Table 2: Anomaly detection performance (%) on CelebA dataset. Standard deviations are computed five different seeds, with results for deterministic cases omitted. The best values shown in bold, and the second-best values are underlined.
color
Eyeglusses
Guidance
Method
AUROC 1
AUPRC +
FPR95
AUROC
AUPRC1
FPR95 +
Baseline
Subset of normal images
ANN
83.3
96.6
62.4
83.0
21.6
47.7
All normal images Anomalous images
KNN
83.4
96.6
62.4
85.3
22.0
43.2
LineurProbe
98.2
99.7 + 0.0
9.6 0.0
99.7 + 0.0
98.4 0.0
0.1
Guide
Language
MCM
84.5
97.2
68.4
3.3
100.0
ZOE
93.9
99.0
35.7
82.6
31.5
67.3
CLIPN-C
82.8 + 1.7
96.8 1 0.4
72.0 + 2.4
1.4 1.1.1
3.7 0.0
100.0 1 0.0
CLIPN-A
84.7 + 1.2
97.2 1 0.3
70.2 2.1
1.2 0.
3.4 0.0
100.0 1 0.0
WinCLIP
93.7
98.9
37.7
83.6
34.6
66.2
Image Language
WinCLIP+
92.8 +: 0.3
98.8 + 0.1
41.2 +: 1.4
85.0 1: 2.4
26.8 3.0
47.8 1: 6.9
InCTRL
85.7 1: 0.9
96.9 1: 0.3
67.8 1: 1.6
87.8 1:: 1.6
30.4 2.99
29.6 1: 4.4
LAFT AD (Ours)
95.0
99.2
29.8
98.1
80.7
5.9
Table 1 presents the main results Colored MNIST dataset with the target attribute being the num- ber. The table is divided into three groups: baseline, guide (Lang,, Img. +angi, and ignore, as discussed in the Preliminaries. The baseline group serves reference point for performance that relies solely on images, as mentioned earlier. Its performance varies depending on the amount of image data available the model. The guide group consists of methods that can be instructed focus on target attribute, where models are given prompts related to the attribute corresponding to the label (e.g., number prompts for number anomalies). Specifically, methods in the guide (Lans) group rely solely on language, while those in the guide (uug. Lang.) group use both image and language to define normality. However, except for our method, language guidance in these methods is used only to calculate image-text similarity and is not applied to image-image similarity. The ignore group rep- resents a method that disregards attributes other than the target, where models are provided prompts of irrelevant attributes (c.g., color prompts for number anomalies). Ignoring irrelevant attributes is unique feature of our method, but is generally more challenging task.
As shown in the table, guided methods (guide (mmg Lmmg)) generally outperform non-guided baselines (kNN with a subset of normal images), with our method achieving the best overall performance. The performance of guided methods that use only language (guide (aan.)) is lower than that of the baseline methods because they are provided with inaccurate prompts for anomalous images (c.g., '13"). This problem, highlighted in Ming et al. (2022), shows that methods relying on image-text
8

---

Published as a conference paper at ICLR 2025
Table 3: Anomaly detection AUROC (%) on MVTec AD and VisA datasets in few-shot settings. We use five different sets of reference samples from the training set. K denotes the number of reference samples. The best values are shown in bold, and the second- -bstt values are underlined.
MVTec
VisA
Medod
=
==1
==2
K 4
==0
==1
==
K
K
IsCTRL
2.
93.2
93.6 +
93.8 13
85.0
86.7 2:7
88.4
89.4
WisCLIP/+
90.4
1.6
95.6 03
96.7 49
85
+
88.2
+ LAFT-- (Durs)
92.6
14
96.1 1:5
96.2 0.3
96.4 0.
80.0
85.0 1.3
86.1 1:3
87.0 1.5
1.1
+ +LAFT-C (Oun)
92.5
94.8
1: 1.4
96.0 1 0.5
96.3 0.4
96.5 0.5
80.6
85.7
1.4
87.0 1.11
87.4
4:
1.4
88.3 =
1.0
similarity in CLIP are highly sensitive to inaccurate prompts. WinCLIP performs similarly to ZOE because sclti-scale features do not benefit semantic anomaly detection. While WinCLIP+ performs better than WInCLIP by using reference images, its performance is still below ours.
The Waterbirds dataset is widely used in studies of spurious correlations and disentangling represen- tations. It consists of two primary attributes: bird type (waterbird / landbird) and background (water / land). Naturally, the training set has a very strong correlation between birds and backgrounds, whereas the test set has an equal ratio of birds to backgrounds. We specify waterbirds and water backgrounds as the normal training set. Table I summarizes the results, where the target attribute is the bird type. The trends observed in the Colored MNIST experiment are largely consistent, demon- strating the applicability of our method to real-works datasets. The key difference that ignoring one attribute (background) does not directly improve performance on the other attribute (bird).
To verify that our method works in multi-attribute datasets, we use the CelebA dataset, which con- tains over 200K celebrity images with 40 attribute labels. For the normal training set, we select two attributes: Hair color and Eyeglasses. The results are displayed in Table 2, where the target attributes are Hair color and Eyeglasses. The trends are consistent with the previous experiments, demonstrating the effectiveness of our method.
The performance of CLIPN and InCTRL was inconsistent across different datasets and target at- tributes, suggesting that their generalization ability is lower than that of CLIP, likely due to the presence of modules trained specific datasets. In contrast, method uses pre-trained CLIP without any additional training, making it more generalizable. Additionally, our method effectively guides one attribute while ignoring the others, as discussed further in the Additional Experiments.
5.2 INDUSTRIAL ANOMALY DETECTION
To demonstrate the practical applicability of our method beyond semantic anomaly detection, we evaluated its performance the widely used MVTec AD (Bergmann et al., 2019) and VisA (Zou et al., 2022) datasets in few-shot settings. However, anomalies in industrial anomaly detection datasets often consist of small defects that are difficult to distinguish using only image-level rep- resentations. Instead of using LAFT AD, which is designed for semantic anomaly detection tasks, propose WinCLIP+LAFT, a model that applies LAFT WinCLIP to extract multi-scale features using CLIP. We apply LAFT to WinCLIP's window, image, and text embeddings, all of which reside in CLIP's shared embedding space, allowing seamless integration.
Typically, some zero-shot or few-shot methods based on CLIP rely on training additional adapter layers transform CLIP's image features for anomaly detection tasks. For example, InCTRL pre- trains feature adapters on specific datasets (such as MVTec AD) to effectively compute the similarity before applying them to different datasets. In contrast, our method uses prompts to transform image features while preserving CLIP'' core features, allowing us to extract features suitable for anomaly detection tasks without the need for additional training of the adapter layer.
For the proof of concept, we used prompts similar to those in WinCLIP for LAFT General (LAFT- G) and more sutegory-specifii prompts for LAFT Category (LAFT-C). For LAFT General, we con- structed prompts using only state words and category names, without providing additional knowl- edge based on the category of the inspection image as in WinCLIP, However, in order to identify a more precise concept subspace, we used more text templates and general state words such as 'mal- formed ()" For LAFT Category, we used prompts that include category-sseciff knowledge (e.g. anomaly class names), such 'bottle with large breakage" for the bottle category.
9

---

Published as a conference paper at ICLR 2025
Table 4: Anomaly detection performance (%) on the Colored MNIST and Waterbirds datasets with various prompts. Standard deviations computed over five different seeds. The best values shown in bold, and the second-best values are underlined.
Concept values
Colored MNIST: Number
Waterbirds: Bird
Prompt
Seen
Unseen
Aux.
AUROCT
AUPRC1 +
FPR95
AUROC +
AUPRC1 f
FPR95
Guide
Only normals
X
96.2
95.7 0.2
94.3
22.6
Partial anomalies
A
X
0.04
98.3 0.0
8.2 0.1
95.5
98.3
18.7
Exact unomalies
98.8 + 0.0
98.8 0.1
6,2 0.1
95,9
98.6
19.6
All candidates
98.5 1 0.0
98.4 1.0.
6.9 0.1
95.6
98.4
20.6
Ignore
Only seen normals
X
x
94.5 0.2
94.1 0.4
25.0 0.5
84.8
92.4
40.6
Partial unseen normall
1.2
95.1 0.3
16.7 004
85.1
92.4
38.1
Exact normals
9..2
06.5
11.6 0.3
85.0
92.2
37,3
All candidates
97.4
96.9 0.2
10.4 0.4
84.8
92.2
38.6
The results are presented in Table 3, which summarizes the average performance across all cate- gories. As shown the results, WinCLIP+LAFT consistently outperforms WinCLIP in both zero- shot and few-shot scenarios. Furthermore, our method achieves superior or comparable performance InCTRL, a pre-trained model for industrial anomaly detection, without requiring additional train- ing. See the Full Results for detailed results on each category.
5.3 ABLATION STUDY ON PROMPT QUALITY
An important consideration when using LAFT is how to provide the user's prior knowledge. In anomaly detection, generally have good understanding of the current training data, but the un- seen test data remains unknown. Therefore, we investigated how the performance of LAFT changes depending on the quality of the concept values provided, as shown in Table In the table, Seen refers to the concept values for the current training data, Unseen refers to the concept values for the unseen test data, and Aux. denotes concept values that are not present in the dataset.
For example, in Colored MNIST, if the guiding attribute is the number, Seen represents the values 0- 4, Unseen corresponds 5-9, and Aux. refers values like 10-20. Similarly, if the ignored attribute is color, Seen includes red, Unseen covers green and blue, and Aux. includes colors such yellow and purple. The symbol indicates that all concept values are used, while A indicates that only half of the concept values are utilized.
The results show that the performance of LAIT is robust to the quality of the concept values when at least partial concept values are provided. Furthermore, the performance is not significantly affected when the completely not included concept values are provided. Also, providing concept values that not included at all does not significantly affect the performance. These characteristies show that LAFT can be effectively used in anomaly detection where only limited information known.
CONCLUSION
In this paper, we introduce Language-Assistee Assisted Feature Transformation (LAFT), a novel feature transformation method designed integrate knowledge and preferences into the anomaly de- tection framework via natural language, without the need for additional data or training. By utilizing the shared embedding space of the vision-language model, LAFT can align visual features with user- provided texts to guide or ignore specific attributes in the image. We also presented LAFT AD, an anomaly detection method that integrates LAFT with kNN classifier, and WinCLIP+LAFT, an extension of WinCLIP that incorporates LAFT for industrial anomaly detection. This combination allows users to adjust the normality boundary of the model by providing texts detect desired anomalies. Through experiments synthetic and real-world datasets, we demonstrate the effec- tiveness of our proposed method.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907 1907.02899, 2019.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Tri Cao, Jiawen Zhu, and Guansong Pang. Anomaly detection under distribution shift. In /EEE/CVE International Conference Computer Vision (ICCV), 2023a.
Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen. Segment any anomaly without training via hybrid prompt regularization. arXiv preprint 1:Xi::2305.10724 2023b.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference Machine Learning (ICML), 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances Neural Information Processing Systems (NeurIPS), 2020b.
Xuhai Chen, Yue Han, and Jiangning Zhang. A zero-/few-shot anomaly classification and segmen- tation method for cvpr 2023 vand workshop challenge tracks 1&2: 1st place on zero-shot ad and 4th place few-shot ad. arXiv preprint 23X5:2335..7382 2023.
Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.023:0 2020.
Niv Cohen, Jonathan Kahana, and Yedid Hoshen. Red panda: Disambiguating image anomaly de- tection by removing nuisance factors. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Ramakrishna Vedantam. Hyperbolic Image-Text Representations. Proceedings of the International Conference on Ma- chine Learning (ICML), 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot dut-tf-diittibutiinn detec- tion based on the pre-trained model clip. In Proceedings of the AAAI Conference on Arlificial Intelligence, 2022.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 17425, 2023.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 2024.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of disto---ist detection. Advances in Neural Information Processing Systems (NeurIPS). 2021.
11

---

Published as a conference paper at ICLR 2025
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1rXiv:88/2.0 246666 2018.
Jeeho Hyun, Sangyun Kim, Giyoung Jeon, Seung Hwan Kim, Kyunghoon Bac, and Byung Jun Kang. Reconpatch: Contrastive patch representation learning for industrial anomaly detection. arXiv preprint arXiv:2305. 2023.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https: otg//0.52// zenodo. 5143773
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winclip: Zero-ffew-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference Machine Learning (ICML), 2021.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http://yannn www.:mleecuccccco.oo wwtp///d/////istt....mmmmm////.. 2010.
Wei-Yu Lee and Yu-Chiang Frank Wang. Learning disentangled feature representations for anomaly detection. In 2020 IEEE International Conference on Image Processing (ICIP), 2020.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Xiaofan Li, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Promptad: Learning prompts with only normal samples for few-shot anomaly detection. arXiv preprint arXIv:2404.05231, 2024.
Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In IEEE/CVF International Conference on Computer Vision (ICCV), 2015,
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint
arXiv:1301.377 1301.3781, 2013.
Yifei Ming and Yixuan Li. How does fine-tuning impact out-of-distribution detection for vision- language models? arXiv preprint arXiv:2306.06048. 2023.
Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
Atsuyuki Miyai, Qing Yu, Go Trie, and Kiyoharu Aizawa. Zero-shot in-distribution detection in multi-object settings using vision-language foundation models. arXiv preprint arXiv:2304.04521, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICMI., 2021.
12

---

Published as a conference paper at ICLR 2025
Tal Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2806-2814, 2021.
Tal Reiss, Niv Cohen, and Yedid Hoshen. No free lunch: The hazards of over-expressive represen- tations in anomaly detection. arXiv preprint arXiv:2306.07284 2023.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müllrr, Klaus- Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:211 1..22//11.111144 2021.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303.15389 2023.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin, Csi: Novelty detection via contrastive
learning on distributionally shifted instances. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In JEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01997,, 2022.
Vitjan Zavrtanik, Matej Kristan, and Danijel Skocaj. Dræm- discriminatively trained reconstruc- tion embedding for surface anomaly detection. In IEEE/CVF International Conference on Com- puter Vision (ICCV), 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-tnnining In /EEE/CVF International Conference on Computer Vision (ICCV), 2023.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu, Unleashing text- to-image diffusion models for visual perception. In /EEE/CVF International Conference on Com- puter Vision (ICCV), 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (IJCV), 2022.
Jiawen Zhu and Guansong Pang. Toward generalist anomaly detection via in-context residual learn- ing with few-shot sample prompts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the- difference sulf-super pre-training for anomaly detection and segmentation. arXiv preprint arXiv:2207.143355 2022.
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND DISCUSSION
Table 5: Anomaly detection performance (%) on the CelebA dataset. We transform the image and text features using LAFT to ignore the Gender attribute. Then, we use the transformed features for the MCM and ZOE methods. Underlined numbers indicate the performance of the suppressed attribute, and bold numbers indicate the performance of the non-suppressed attribute.
Hair color
Gender
Method
AUROC t
AUPRC1
PPR95 4
AUROC f
AUPRC1
PPR95
MCM
88.0
97.9
56.7
96.9
98.0
13.7
+ LAFT
93.8
99.0
35,4
32.4
50.0
98.4
ZOE
93.4
98.9
38.8
99.4
99.6
02.5
LAFT
95.1
99.2
30.6
94.7
Ignore attributes using LAFT Unlike in a simple Colored MNIST dataset, we observe that ig- noring one attribute using LAFT does not directly improve the performance of the other attribute in real-world datasets. However, as seen in Appendix C, the LAFT actually suppresses the attribute to be ignored. As mentioned in the Preliminaries, in the embedding space using only text prompts. Alternatively, guiding the attribute is relatively easy, because LAFT only needs capture the primary information about the attribute.
hard remove all attribute-relatd information
Selecting the number of PCA component d LAFT introduces hyperparameter d, which rep- resents the number of PCA components. While the performance remains relatively stable within a certain range, as demonstrated in Appendix C. selecting an appropriate value remains crucial. How- ever, determining the optimal d is particularly challenging in the context of unsupervised anomaly detection, where ground truth labels are unavailable. Consequently, must be selected heuristi- cally. Developing an automatic selection mechanism for d would be valuable direction for future research to improve the usability of L.FT.
Using LAFT with other methods LAFT can be used not only for anomaly detection, but also as feature transformation module in other tasks or methods. Basically, we expect that it can be applied any vision model that requires a feature extractor. For a simple example, we apply the LAFT method to MCM and ZOE. The results in Table 5 show that we can suppress the Gender attribute. Applying LAFT other downstream tasks would be a future work.
B EXPERIMENTAL DETAILS
We provide the source code of our method at httpp:://ggttieie com. hwtp://gg///////////
Backbones For semantic anomaly detection datasets (Colored MNIST, Waterbirds, CelebA), we the CLIP ViT-B/16 (Radford et al., 2021) with the pre-trained checkpoint from Fang et al. (2023). And since InCTRL (Zhu & Pang, 2024) and CLIPN (Wang et al., 2023) require pretrained weights, used the network with the pretrained weights provided by the authors. For industrial anomaly detection datasets (MVTTe AD), we use the CLIP ViT-B/16+ (Gadre et al., 2024), pre-trained on the LAION-400M (Schuhmann et al., 2021) dataset, following the setup used in WinCLIP (Jeong al., 2023). For a fair comparison, we also adopted the CLIP's image encoder as a feature extractor for the &NN baseline.
Metrics We three metrics evaluate the performance of the methods: the Area Under Re- ceiver Operating Characteristics (AUROC), the Area Under the Precision Recall Curve (AUPRC), and the False Positive Rate at the 95% true positive rate (FPR95). AUROC and FPR95 are com- monly used for the anomaly detection or out-of-distribution detection task (Ming et al., 2022). And we also use AUPRC because some datasets are imbalanced, with significant disparity.
Computing resources
We use single NVIDIA RTX 3090 GPU for all experiments.
14

---

Published as a conference paper at ICLR 2025
Hyperparameter The only hyperparameter in LAFT is the number of PCA components d. We typically choose from 4 to 32 when guiding an attribute and from 32 to 384 when igmoring an attribute. Refer to the Ablation Study for the impact of d on the performance. And we use & 30 for the methods using KNN anomaly scoring (kNN and LAFT AD).
Prompts To the actual prompts used in the experiments, please refer the laft Laft/prompts in the source code. We referenced the prompts provided by CLIP (Radford et al., 2021)
Colored MNIST
Number: "zero", ..... "twenty". "O"..... "20"
Color: "red", "green", "blue", "yellow", "orange"... ..... "black", "white"
Waterbirds
Bird: Class names provided by the dataset and Birdsnap class names,
Color: "land", "bamboo", "forest", "ocean", and similar words.
CelebA
Hair color: "blond", "black", "brown", "gray", "red", "white", and similar words.
- Eyeglasses: "glasses", "eyeglasses", "sunglasses"
Gender: "man", "male", "boy", "woman", "female", "girl", "masculine", "feminine"
MVTec AD and VisA We use the prompts WinCLIP (Jeong al., 2023) for anomaly scoring. To compute the LAFT concept subspace, we use some more template-Istate--eve prompts for both LAFT General and LAFT Category. For LAFT Category, use additional category-level prompts as Li et al. (2024).
Template-level: Jeong et al. (2023) and "an image of ()", "a photo the {}",... State-level: Jeong et al. (2023) and "O in perfect condition", "malformed O".... Category-level (LAFT Category): "bottle with large breakage", "carpet with hole".
Dataset Split
Colored MNIST denotes red, denotes green, and B denotes blue colored digits. 0-4 and 5-9 denote the digits from 0 to and from to 9, respectively.
TTain:: R/0-4 (16.67%)
Test: R/C-4 (16.67%), R/5-9 (16.67%), 0-/0-4 (33.33%), GB/5-9 (33.33%)
Wbird denotes waterbirds, and Lbird denotes landbirds. Wback denotes water background, and Lback denotes land background.
Waterbirds
-Train: Wbird/Wback (22.04%)
Test: Wback (1 (11.08%), Wbird/Lback (11.08%), Abird/Wback (38.92%),
abbrd/Lback (38.92%)
Blond denotes blond hair, and Glass denotes eyeglasses. -Blond denotes non- blond hair, and -Glass denotes no eyeglasses.
CelebA
-
TTrinn: B1 ond ass (14.66%)
Test: Blond/Glase (13.01%), Blond/-Glass (0.31%), -Blond/Glass (80.53%) -Blond/-Glas (6.15%)
-
MVTec AD and VisA
We use the same split as Bergmann et al. (2019) and Zou et al. (2022).
https://github..oo htt.:ppaaaaaiiiii..pp http:///gnannncccl//////////e//a
15

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
C.1 GUIDING AND IGNORING ATTRIBUTES
Table 6: Anomaly detection performance (%) the Colored MNIST datasets with different target criteria. We use five different seeds to split the training set for coloring each digit. Bold numbers indicate that the performance should be high (relevant), and underlined numbers indicate that the performance should be low (irrelevant).
Number
Color
Criteria
AUROC
A
FPR95
AUROC
AUPRC+
IPR95
No guidance
ANN
89.7 + 0.4
89.5 0.4
44.0 0.3
81.1 0.6
89.5 1.5
58.1 0.2
Guide
Number
98.5 + 0.0
98.4 + 0.0
06.9 +
52.7 0.1 0.1
666.4 + 0.2
89.3 + 0.2
Color
+..2 0.1
53.6 +
94.8 + 0.1
100.0 + 0.0
100.0 - 0.0
0.0 + 0.0
Ignore
Number
63.7 13.7:00.. 0.2
65.6 0.2
75.8 + 0.1
99.9 0.0
99.9 + 0.0
0.2 0.0
Color
9..4
96.9 + 0.2
10.4 + 0.4
60.2 + 0.4
73.0 + 0.5
78.0 + 0.1
Table 7: Anomaly detection performance (%) on the Waterbirds dataset. Standard deviations are not reported because the method is deterministic. Bold numbers indicate that the performance should be high (relevant), and underlined numbers indicate that the performance should be low (irrelevant).
Bird
Background
Criteria
AUROCT t
AUPRC
FPR95
AUROC f
AUPRC
FPR95
guidance
ANN
82.3
91.2
48.1
68.0
63.9
87.3
Guide
95.3 59.5
98.2
19.5
70.6
72.6
87.2
Background
84.5
91.4
97.6
97.2
10.1
Ignore
Bird
63.9
59.3
87.2
74.5
87.6
62.6
Background
84.8
92.2
38.6
52.7
50.0
92.6
The results in Table 6 and Table show the performance of two attributes when one attribute is either guided or ignored. In datasets with two clearly distinguishable attributes, such Colored MNIST, ignoring one attribute implicitly guides the other. However, datasets composed of real- world images, such as Waterbirds, although there may appear to be only two attributes, there may actually be many more. As result, while the performance non-ignored attributes may improve slightly, the overall improvement is not significant. Nevertheless, when an attribute is ignored, can confirm that it is indeed properly disregarded.
C.2 ABLATION STUDY ON PCA COMPONENTS
To investigate the impact of the number of PCA components on the performance of LAFT, we con- duct an ablation study on semantic anomaly datasets. The results are shown in Table 8 and Table 9. We observe that the performance LAFT's not very sensitive the number of PCA components for certain range of d. However, the performance drops significantly when d is too small or too large. This because a small cannot capture the concept subspace well, while large d may include irrelevant information. Except for Eyeglasses attribute in CelebA, the best performance is achieved when between 14 and 28 for guiding attributes. For the Eyeglasses attribute, the best performance is when d is 6, which expect because is simple binary classification problem of whether person wears glasses or not, rather than distinguishing between different types within an attribute (like distinguishing between different numbers or different types of birds).
16

---

Published as conference paper at ICLR 2025
Table 8: Anomaly detection performance (%) on Colored MNIST and Waterbirds datasets. We scan the number PCA components d, which the only hyperparameter of LAFT module. The best values are shown in bold, and the second-best values underlined.
Colored MNIST: Number
Waterbirds: Bird
d
AUROC
FPR95 AUPRC
AUROC f
AUPRC f
FPR95 +
Guide
80.3
79.5
73.7
87.2
95.0
67.7
93.5
94.2
42.3
91.2
95.9
35.7
6
96.3
96.3
19.3
93.6
97.8
26.4
8
97.2
97.3
14.7
93.9
97.8
25.5
10
97.4
97.3
13.2
94.2
97.9
23.8
12
97.7
97.7
11.1
94.6
98.1
22.4
14
97.8
97.7
10.0
95.1
98.2
21.8
16
98.0
97.7
10.5
95.4
98.2
21.0
18
98.2
97.8
9.0
95.6
98,4
20.6
20
98.4
97.9
8.8
95.6
98.3
20.3
24
98.5
98.2
8.4
95.5
98.2
20.1
28
98.4
98.1
7.9
95.4
98.2
19.3
98.3
98.0
95.0
98.0
21.3
97.8
97.6
10.5
94.9
97.9
21.1
97.7
97.5
10.7
94.4
97.6
21.4
97.1
96.8
13.1
93.5
97.1
23.0
Ignore
8
96.2
95.3
16.2
82.6
91.1
41.7
16
96.8
95.9
13.3
82.6
91.1
42.0
32
97.1
96.2
12.0
83.4
91.4
40.8
64
97.1
96.4
11.9
83.4
91.4
40.1
96
97.2
96.4
11.8
83.5
91.4
40.2
128
97.3
96.6
11.0
83.4
91.4
40.1
160
97.4
96.7
11.0
83.4
91.4
40.0
97.4
96.8
11.0
84.8
92.2
38.6
224
97.4
96.9
10.6
83.8
91.7
41.8
97.2
96.7
11.9
83.8
91.7
42.4
288
97.1
96.6
12.5
82.3
91.0
44.0
320
96.9
96.3
13.6
81.6
90.6
45.6
352
96.7
96.1
15.1
81.6
90.7
44.6
384
96.4
96.1
17.0
81.5
91.0
46.4
Table 9: Anomaly detection performance (%) on CelebA dataset. We scan the number of PCA components d, which is the only hyperparameter of LAFT module. The best values shown in bold, and the second-best values are underlined.
Hair color
Eyeglasses
d
AUROC 1 AUPRC
FPR95 4
AUROC +
AUPRC f
FPR95 J.
Guide
2
52.6
88.4
95.3
65.9
11.4
86.3
4
90.5
98.3
55.8
97.3
76.4
8.2
92.6
98.7
38.4
98.1
80.7
94.0
99.0
36.5
97.9
78.6
94.6
99.1
31.9
97.5
75.0
8.3
94.8
99.1
31.7
97.6
77.0
95.0
99,2
29,8
97.6
78.3
94.8
99.1
30.6
97.3
75.3
11.1
18
94.7
99.
29,5
96.8
72.0
13.5
94.6
99.1
30.5
95.3
60.1
18.1
94.3
99.1
31.7
95.2
57.1
18.6
28
94.0
99.0
33,9
94.8
55.7
21.0
32
93.9
99.0
34.4
94.2
47.4
20.6
40
93.3
98.9
38.0
92.5
40.6
24.5
48
92.8
98.8
38.5
91.4
38.4
28.5
64
91.8
98.6
40.9
89.7
32.7
31.8
17

---

Published as a conference paper at ICLR 2025
C.3 LAFT WITH VARIOUS VISON-LANGUAGE MODELS
Anomaly detection AUROC (%) on semantic anomaly datasets. We compare vision- language models across diverse backbone architectures and training strategies. The best values shown in bold, and the second-best values are underlined.
Table 10:
CLIP VIT
CLIP ConvNeXt
EVA02
SigLIP
VIT
CoCa VIT
Method
B-16
L-14
H-14
Base
XXL.
B-
L.14
B-16
1.16
B-32
L-14
Colored MNIST: Number
KNN
92.4
90.7
89.8
84.2
87.5
66.7
73.1
86.1
82.3
85.4
89.4
MCM
62.9
79.8
87.1
73.9
83.9
40.2
64.0
74.7
68.2
62.1
70.4
ZOE
91.2
90.7
93.3
92.7
96.2
83.4
92.7
93.0
95.9
93.6
97.4
LAFT AD (Ours)
98.5
98.9
99.6
97.6
99.5
89.0
94.3
98.5
99.1
98.7
98.8
Waterbirds: Bird
ANN
82.3
87.1
85.9
79.7
85.9
86.0
88.7
83.6
79.2
76.0
84.8
MCM
88.8
94.3
94.7
85.2
92.6
88.8
94.9
89.7
93.8
81.5
87.4
ZOE
92.2
93.8
93.77
90.6
93.5
93.6
94.4
93.3
94.2
88.4
92.5
LAFT AD (Ouns)
95.6
97.2
97.3
94.4
97.2
96.6
98.6
96.2
97.6
91.1
95.4
CelehA: Hair color
RNN
83.3
85.3
83.7
88.0
88.3
86.0
85.4
90.4
89.2
84.5
N6.7
ZOE
93.9
94.3
93.5
94.1
90.1
LAFT AD (Ours)
95.0
94.8
95.1
94.4
95.1
94.9
94.5
94.8
94.8
94.8
93.7
CelebA: Eyeglasses
RNN
83.0
81.0
75.0
77.5
78.1
76.2
75.5
75.8
70.0
79.6
78.1
MCM
5.7
11.7
13.2
12.6
11.5
21.6
24.8
12.2
14.9
10.6
13.6
ZOE
82.6
98.8
98.9
94.8
93.9
98.7
97.9
99.2
84.9
91.1
LAFT AD (Ours)
98.1
97.9
98.1
98.1
97.9
97.6
28.1
98.1
98.1
97.6
We compare the performance LAFT with vision-language models across diverse backbone archi- tectures and training strategies on the semantic anomaly datasets. We use architectures with pre- trained weights available OpenCIIP (Ihharco al., 2021), I). Specifically, we use ViT (Dosovitskiy al., 2021), l), ConvNeXt (Liu et al., 2022), and EVA02 (Fang al., 2024) as the backbone architec- tures, and we use the pre-trained weights from CLIP (Radford et al., 2021), EVA-CLIP (Sun et al., 2023), SigLIP (Zhai et al., 2023), and CoCa (Yu et al., 2022). The results shown in Table 10. We observed that across various vision-language models, LAFT AD consistently outperformed the baseline method in most cases.
D FULL RESULTS ON MVTEC AD AND VISA
We provide the complete results on the MVTec AD and VisA datasets in Table 11 and Table 12. As shown in these tables, LAFT significantly enhances the performance of both anomaly detection and localization in the zero setting. Additionally, LAFT improves anomaly detection performance in the few-shot setting.
Despite these improvements, there certain limitations to applying LAFT to industrial anomaly detection datasets. First, as discussed in the Limitations, the selection of d relies on a heuristic approach, requiring manual tuning for each category. Second, LAFT does not improve anomaly localization performance. We hypothesize that this is due to the localization task requiring more fine-grained information than anomaly detection, while LAFT may remove subtle details that are not well captured by text prompts. Nonetheless, even in such cases, our method achieves anomaly localization performance that remains comparable to the original WinCLIP.
Developing more sophisticated approaches for applying LAFT to industrial anomaly detection mains an important direction for future research.
18

---

Published as a conference paper at ICLR 2025
Table 11: Full anomaly detection and localization AUROC (%) on the MVTec AD dataset in few-
shot settings (k-shot). We use five different sets of reference samples from the training set for each
method. We bold the best performent and enderline the second-best performance in each
caleea Sategory
Average
91.3
93.5 16
94.2 14
94.8 14
9
9522207
11.5
98.0 1.0.
93
95
96.2
96.3 96.3
93.8 1.3
95.7:0000
96.4 #00
96.5
20
Average
93.7 03.7
23.1 #003
92.5
94.3
93.5
9:
94.6
94.3
94.
94.+
24.5022
24.50000
90.4
92.6
92.5
96.5
82.3
85.2
88.5
Zupper
039=08
9
95.1 =07
=7
Zapper
96.7 96.7 =02
95.3 =05
974=011
=01
970
2.1
97.1
977 =01
97.
=01
97.--0
-01
978 978=01
978=01
91.2
91.
92.5
88.8
92.6
94.7
949
939
95.7
9699
97A
94.2
97.0
97.4
96.2
97.
97.8
91.7
92.0
942
97.4
97.
975
97.
97.
Wood
99.3
99.3
99.3
97.
9944 99401
99.3 =3
99400
97.6-2
99
-9.3=11
99-201
Wood
00.5
25.3 =0
93
94.7 204
95.4
00
00
95.5
95.5
94.-022
98.6-11
95.5
98.
97.0
99.3
99.3
99.4
85.1
89.9
88.3
95.5
Transstoor
91.2
2:03
91.3
834
19415
91.7
1.02
87.2
90.4
92.0-00
92.0
88.8
91.1:000
02.2202
2003
Trunsistor
82.2 #
89.3 3.33
86.8 1:66
89.3 10.5
19.7 N9.7
1.1
85.0-022
90.2
89.
91,2
88.8
40.
89,
9
20.9
81.
10.0
91.5
92.2
83.2
86.7
83.5
Tboihbeusi
26.2
05.044
9..1
96.2 26.2433
97.8
96.911
97.7
97.7
98.7
98.2
98.7
94.944
28.3
Toolbbuuh
96.2 06
25.4 03
95
96.905
96.0
95
98.2
97.4
97.2
98.5+
970+
97.+
01
85.3
87.2
87.5
98.4
$6.2
89.6
88.
99.9 999000
99.9
100. =00
99.9 #00
80.
91.6
94.
24.02033.
91.810.11
95.00.11
91.110.11
91.8
95.0
91.8
949
Tile
999 9
Tile
94.2
99.9
99.9
99.9
99.9
100.
99.9
100.
999
9999
999
100.
79,1
87.5
85.9
98.1
Scrrw
74.3 74.3
77.9 21
22.2
83.1 =
77.0
78.3
ML.7 2.5
A 1
84.6 22
=1
Scrow
96.1 :01
203
=06
202
103
945
201
96
#02
#03
965#22
#02
97.0
-004
003
00
71,6
78.6
22.1
81.1
N2.7
81.0
K3.0
81.2
848
91.1
819
28.4
94.4
80.6
96.4
94.6
94.5
966
94.7
965
97.0
96.8
96.9
91.3 1.2
=03
=02
=03
93.8 002
=02
=0
=02
0.
0.
02
02
002
Pill
86.3
%6.4
80.0 19.8-2
93.3
9288
92.5
93.6
93.2
93.0
912-
93.5
93.2
93.0
PM
72.7
87.1
85.2
94.1
94.3
94.1
94.4
94.5
94.2
94.5
94.
94.3
24
946
Metal Noi
95.3 4.5
08.6000
03
04
95.5
9902222
99.2 0
99.2 + 02
22.3 22.3
99.3
99.1
19.1
Metal
18.11111 1.1
73.3
0.9
24.4
1.0
794
1.
74.3
L.
75
81.3
76.1
722
#2.3
77.0
06
78.1
066
96.22
97.1
97.1
98.
98
99.1
99.0
97.1
99.2
49.3
51.2
$2.2
78.
78.1
Leaiher
99.9 #01
100. 200
100
100. 2:0
99.9
100
100
99.9 #01
400
200
200
200
Leather
00000:00
29.22
000
29.2
::0
99.3 1:00
99.2 99.2.100
1100
99.2 99.21000
1:00
99.3 #.00
99.2 99.2
#00
99.2 99.2 #:00
99.3 +0.0
99.3
99.2 99.2
+00
100.
100.
100.
100.
100.
100.
100.
100.
100.
95.5
96.3
96.3
99.3
29.2
29.2
Haueltuu
92.4
92.4
92.4
97.422
96.8
00
96.8
96.
96.9
09
97.2244
97.6
97.7 97.7100
97.7 97.7
98.2 =
97.9 779=002
98.0 =
97.5 1.5=02
=0
97.8 =0
97.9
=0=22
Handlou
95.2
95.6
95.9
9 03
98.5
08.3002
03
08.3
0
28.3
=2
98.7 0.1
98.6 0.1
98.6200 0.1
98.6
98.8
98.6
01
98.6 01
98.8
28.6
08.6
01
Grdd
99.5
900
19.1100
401
201
Grid
94.3 # 10
94.7 #:07
94.1 #:08
9499 1:09
95.0 1003
95.1 1:06
95.4
9:04
#:04
95.1 4:00
404
978 401
95.3
99.2
99.3
99.4
98.8
99.4
99.6
99.5
99.5
99.6
98.8
99.3
99.4
79.4
82.0
82,3
97.6
25.5
Carpet
99.4
90.6
9:01
9:.1
998104
99.10.
99.8
99640.3
9966022
909
99.8 4.0.1
20.
9..8
9:0000
900000
Curpet
99.0 101
99.0
98.9
989101
90.0101
989
98.
98.
98.
98.9
98.9
#01
99.3
99.8
99.8 8
99.8
90.9
94.7
94.4
99.0
Capslle
1.0
1.9
292252
85.9233
120
133119
75.9
87.1
94.7
74.7
946200
Capsale
95.6 103
96.1200
100
10.1
10.1
10.1
97.0
20.2
97.3
27.
+0.3
95.8
97.4
68.7
20.2
$1.3
68.8
72.7
27.1
78.3
93.3
23.5
$7.0
84.9
91.2
95.6
25.2
96.1
95.8
Cabllc
61.1 21
1007
1444
89.7 1:06
177
103110
1.07
90.5 1:08
#:0
107
Cable
89.8::0
1:08
81.0 03
85.0
1:
90.5 1:07
8099 1.03
86611:0
1:07
90.8
4:04
87.6
1:05
1:03 868
91.1+-03
82.8 +:04
3694404
1044
K5.1
89.6
89.7
90.3
918
899
90.7
91.2
92.1
91.4
91.3
61.3
68.3
66.7
89.8
85.0
8666
90.8
87.6
91.1
869
Bottle
98.6
98.7
98.7
98.8
99.4 19.4..
99.4 49.4...
99.6
99.6 1660
99
99.5
4
98.2
97.7+2.5
9.1
99.7 0.1
Botlee
85.7
90.0
90.1
94.8 0:0
96.2 0.1
96.2 06.1 0.1
9.00
96.4 0.0.
96.4 0..1 0.1
95.1 0..1
0.1
96.5 0..1
0.1
96.5 4 0.1
95.2 0.1
96.6
96.6
011
LAFT-C (Ours)
LAPI-G (Otm))
LAFT-C (Oirs)
LAFT-C (Ours)
LAPT-G (Oua))
C (Ours)
LAPI-G (Diss)
LLPIICC (Durs)
LAFT-G (Ous)
+ -LAF--C (Durs)
+ LAFT-G (Oie))
+ LAFT-C (Aur--CO0r)) (Ours)
+ LAFT-G (Ome)
+ LAFT-C (AuF--COUr)) (Ours)
LAFT- (Ons))
+ LAFTC (Qur)) (Ours)
+ LAFT-G (Ous))
+ LLFF-C ((ur)) (Bur))
Anomaly Localization
Anomaly Detection
+ LAFT-G
LAFT-G
Method
WinCLI
InCTRI
WiICH JP
INCERL
WHnCLIP
INCTRL
WINCLIP
INCTRL
WInCLIP
*K Method
WinCLIP
WHOCLIP
2 WHOCLIP
4 WinCLIP
WinCLIP
#K
19

---

Published as a conference paper at ICLR 2025
Table 12: Full anomaly detection and localization AUROC (%) on the VisA dataset in few-shot
settings (k-shot). We use five different sets of reference samples from the training set for each method. We bold the best performance and underline the second-best performance in each category.
Average
85.7 11
86.7 2:22
85.6
+18
1.1
87.0 11.1
88.4 2.0
86.8 1.9
#1.05555
87.4 1
89.4 1.7
88.2 = 1.2
68.2 2.
88.5 &:
1.0
Average
005
23.0000
94.8 1.
93.7 2.03
93.3 203
95.0 =03
93.8 003
93.9
95.2 t
93.9 =0
94.5
80.8
85.6
86,
87.0
88.2
Pipe Fryum
890 13
97.4 1.3
222 +16
90. =08
900 =15
98.5 =03
92.3 16
90.5 115
89.6 155
98.1 =0.1
926
91.4 2.4
25.4 004
95.9 =04
9588 =.3
95.4 =00
9588 =033
95.8 =03
95.5 =03
958 =03
02
95.7 02
=02
92.1 =
910
989
70.6
74.6
899
Pipe
80.5
85.9
95.9
Pebsi
=
59.0 -
88.026
53
81.6 81.6
= 31
=03#3.
88
92.8 33
22.0 29
934
954=
16
96.1 13
144
Pelli
95.7 =3
24.0 =00
95.4
96.1 =:02
332201
20.0004
96.3 03
95.3
96.7 02
96.8
02
96.
86.
91.2
20.3
96.2
91.4
95.2
76.1 2.0
#:3
-31
=52
20.2 221
84.2 1
2.2
#24
81.3 #23
828 1.9
N0.5 2.3
1.9
848=08
206
855
288
203
94.7 203
244 003
926624
Pe53
62.
67.8
68
70.8
70.8
81.7
74.8
725
72.5
80.4
78.0
85.9
83.1
Peha
73.2
20.8
36.6
934
93.1
93.1
94.1
92.0
Peb2
89.1
56.2
748
61.4 31
290
642226
2-20
77.9
64.2
64.2
2-
6164211
64
73
6.9
63
M0.1
11
67.61 1.2
68.108
65.9
0.6
Peh2
50.8
7R.8
91.5
89.4
094
92.104
92.14..3
90.0
926403
22220
904403
93.1
93.3
91.0
93.5
Pchl
76.8
84.1
203
844
8I.1
491
83.9
+08
84D
203
86.9
11
86.9
85.
86.4
22
25
89.
89
134
90.0
1000
24
Pchl
41.5
79.0
96.2
94.
#04
96.3
#03
408
96.8000
954402
970
203
96.3
#07
975
97.2
98.0006
Macuroni2
79.0 2-4
72.
122
730212
1:29
76.4
+009
74.6
74.6
81.7 45
77.0
7522
75.2
249
76.4
224
74.12.
74.6 2:22
Macuran2
21.3
91.2
131
914
9122
929
211
900
92
20.5
904
65.2
63.5
62.5
24.5
MA.1
26.4
53.5
27.6
79.9
2111
20.5
Macuroni
834
85.
14
MSS L1
87.9 1:
849
86. 1
86.3 20R
88.022
84.0
84.8
89.5
1:0
84
85.01044
84.3
Macuroni
93.
21.4
999
93.7
918
90.3
9A1105
910003
89.6
03.0
90.4
229
74
80.0
57.4
84.4
B8.6
Fryum
1.
1:88
95.3 1:0.3
91.6 4-09
93.2
#: 1.0
234 #1 1.0
949
#.08
91.9 1:08
92.6 1.15
23.4 1
95.3 20.
207
92.4 1.08
93.2 1.4
92.7 1.0
Fryum
#:09
918 1.04
4.02
#05
92.1 #02
#01
1:03
92.4
1:02
92.5 101
202
84.3
N2.5
91.8 81.824
V1.9
23.4
949
82.1
87.6
89.7
22.4
93.0
92.5
940
92.5
94.3
92.4
948
Chew Chewinggum
=
=0
99.2 02
98. =
97.3 =:04
98.7 -0-0001
99.2 =2
99.0 00 01
97.5 03
98.6 6.6=20 02
99.3 =
29.0 0.
97.2 97.7=00 03
98.7 03
994
99.0 02
Chew Chew-inggum
28.6 08.6=00 00
99.1 =0
99.1 0:00
98.6 00000:00
0:0
99.1 00.00000
99.1 =00
98.6 0806=00
98.7 =:0
99.1 00
28.3=00 00
99.1 =00
95.9
97.8
98.00
95.1
97.0
95.4
28.3
Cubbe
=
946
946104
92 1.6
94.4
948
9.
92.9
95.1 0.5
95.5 0.4
# 0.4
92.2
95.2 0.8
95.5.006 0.6
95.5
0.7
Cashew
96.5
25.2
25.2 0.5
96.6
0.2
95.34002
9..2
96.7
95.2
1: 0.1
96.8
2 0.3
# 0.3
9.. 01
93.3
95.4
96.6
95.4
95.3
Capssles
123-212
848213
14
78.9 111
+39+1.0
11.0
209
80.3 1.4
11
872#111
L1
21.
858
885100
1.06
2055
Capsales
=04
#03
102
203
93.7 203
883
203
100
93.9 #03
88. =03
879
79.1
53.3
84.1
77.1
82.3
86.5
859
87.7
8722
885
87.9
70.0
72.7
20.3
93.2
98.0
87.7
B83
879
83.3
879
Candle
92.2
97.5 033
97.5 1.333
85.0 1-4.
92.5 4-02
97.8 02
97.8 97.8010
0:01
00.5 08.5 L.5
97.7 0:02
98.0 + 02
97.9 02
89. 108
17.90:00 0.
98.1 1.0.1
98.1 0.
Candle
26.0 +:0
26.0++03.
96.5 1.02
95.9 1.02
96.0 0:02
96.621
9600
96.2 06.24:0.11 1.0.1
96.7 4..1 0.11
96.2 96.2+ 011
96.2 01
90.4
90.0
96.5
96
LAPT:: (Dun)
LAFTEC (Oun)
LAFT-G (Ouu))
+ LAFI-C (0us)
LAFT-G (Ous)
+ LAFTC (Omn))
+ LAFTAG (Ous)
LAFTC (Oun))
Anomaly Localization
LAFT-C (Oun)
LAFT-G (Dus)
LAFF-C (Oun)
LAFFG (Ous)
LAFTE (Ous)
LAFT-G (mm))
LAFFCC (0mm)
LAFEG (Uu))
LAFRC (0mn)
Anomaty Detection
Melhod
WiOCLIP
LAFT-G
LAFT-C
InCTRL
WHOCLIP
LaCTRL
WiOCLIP
LOCTRL
WisCLIP
TRI
WiOCLIP
Method
WOCCLIP
LAFT-G
WiCCLIP
WisCLIP
WOOCLIP
8 WiaCLIP
#K
2
#K
2
20

---

Guide Published as a conference paper at ICLR 2025
E ALGORITHM
The following pseudocode demonstrates the implementation of LAFT AD, using syntax similar to NumPy, as the notation used in (Radford et al., 2021).
model: the CLIP model
prompts: the 1ist of prompts provided by the train images: the collection of normal images test jest_images:: the collection of images to be tested d: the number of principle axis
Compute attribute subspace text features model. text text (prompts) pair_diffs basis = pca (pair_diffs, d)
ttext_featuree
Encode images train_featuree gadel.eencdd_mmeeee (crain_imager
test_features model .ecaadeencceeiiiei image (tss__images)
train laft features inner cumer_projectionntraa..e features, basis)
test last__aft_feealt features = imer_prooeetionnteet__eee
basis)
anomaly_scores knn lttrrinnnafttffetuurss test_lafttfeattres)
Ignore = cethogonal__rojjcttinn (train_features, basis)
test lal_llattffaarrre = orthogoonal_projeeetoonn (test_features) basis)
anomaly_scores knn [tttarai_lft___fatuuue test_laftt
21


Reviews:
[{"review_id": "YclN59v2Np", "reviewer": "Reviewer_P8a7", "summary": "The paper introduces a feature transformation method aimed at focusing on specific image attributes guided by language. The approach, termed Language-Assisted Feature Transformation (LAFT), leverages the shared embedding space of vision-language models (specifically CLIP) to modify image features according to user-defined concepts expressed in natural language, enabling enhanced anomaly detection capabilities without additional training.", "questions": "1. What does $c_i$ represent in Equations 5 and 6?\n2. For zero-shot anomaly detection, can the transformed image features still match the text features effectively?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors explore a valuable research topic that contributes to the current body of knowledge—how to adjust decision boundaries using language to enhance CLIP’s anomaly detection performance. \n- The proposed method stands out due to its training-free nature, which provides flexibility in application across various tasks with limited data.", "weaknesses": "- The paper uses the vector difference between two textual descriptions to represent a single attribute and maps this attribute directly to image feature transformation. However, this simplification raises at least three issues:\n   - The properties of objects cannot be adequately represented by the difference between two concepts.\n   - Real-world attributes are often complex and may involve different colors or textures across various parts of an object.\n   - The text embedding space and the image embedding space in CLIP are not perfectly aligned; therefore, vectors derived from the text space may not be directly applicable to the image space.\n\n- To validate the effectiveness of feature transformation, using a CLIP-based classification task would be more suitable than anomaly detection.\n\n- The paper lacks results on anomaly localization, which is crucial for industrial applications.\n\n- The language throughout the paper could be clearer. It is recommended to refer to previous works using proper method names and provide concise descriptions of these methods.\n\n- The axis labels in Figure 3 are inconsistent. How were the attributes 'Number' and 'Color' derived?\n\n- The dataset chosen for experiments, SEMANTIC ANOMALY DETECTION, focuses on distinguishing simple concepts. Why not test the method on widely recognized OOD datasets such as ImageNet-1k and OpenOOD? Industrial anomaly detection would benefit from validation on datasets like VisA and Real-IAD as well.\n\n- The comparison methods included are relatively weak. Why not compare with more recent OOD detection approaches such as NegLabel [1] and ClipN [2]?\n---\n- \\[1] X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han, “Negative label guided OOD detection with pretrained vision-language models,” in The Twelfth International Conference on Learning Representations, 2024.\n- \\[2] Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. ClipN for zero-shot OOD detection: Teaching CLIP to say no. ICCV, 2023.\n---\nIf the author can address my concerns, I will consider increasing the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6S1EoEgT9", "reviewer": "Reviewer_DtcB", "summary": "The paper introduces Language-Assisted Feature Transformation (LAFT), a novel framework that leverages vision-language models (like CLIP) to enhance anomaly detection. Traditional anomaly detection methods often struggle to capture user-defined nuances of normality, particularly when attributes are entangled or datasets are incomplete. LAFT tackles this by enabling feature transformations guided by natural language prompts. These prompts align visual features with user intent by projecting image features onto specific concept subspaces within a shared embedding space. The paper also proposes LAFT AD, a k-nearest-neighbor (kNN)-based method combining LAFT with anomaly detection, and extends this work into WinCLIP+LAFT, designed for industrial applications. The effectiveness of LAFT is demonstrated across datasets like Colored MNIST, Waterbirds, CelebA, and MVTec AD, showing superior performance in both semantic and industrial anomaly detection.", "questions": "1. In Table 8, the header refers to \"bird,\" which is inconsistent with the title of the Colored MNIST dataset mentioned (maybe a typo). Could the authors clarify this discrepancy?\n2. What are the sizes of the training sets for each dataset used in the experiments? Given that these samples serve as candidates for kNN search, how might the number of training samples affect the final performance of the model?\n3. The experimental results on the MVTec AD dataset in Table 3 suggest that InCTRL might outperform WinCLIP+LAFT when considering deviation, especially when the number of shots exceeds 2. Could the authors provide detailed experimental results for each of the five different reference sample sets?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. LAFT bridges a gap in anomaly detection by allowing users to express preferences using natural language, providing more control over what is considered \"normal.\"\n2. Unlike other feature transformation models, LAFT does not require additional training, making it efficient for settings with scarce data.\n3. The experimental results demonstrate that LAFT outperforms state-of-the-art methods, particularly in semantic anomaly detection tasks.", "weaknesses": "1. While LAFT demonstrates significant improvements in controlled environments, such as the Colored MNIST dataset, its performance gains appear less pronounced when applied to complex real-world datasets. This discrepancy suggests that the model may struggle to maintain robustness across multiple intricate attributes, highlighting the need for further refinement in handling multi-attribute scenarios.\n2. The experimental setup lacks comprehensive comparisons, particularly between language-assisted and vision-assisted approaches. For instance, incorporating image guidance by utilizing related reference normal images (e.g., normal digits in various colors) or color-augmentation for kNN baseline could provide valuable insights. A thorough examination of both language-based and vision-based assistance would strengthen the evaluation of LAFT's efficacy.\n3. The impact of the number of PCA components, which is the sole hyperparameter in LAFT, is not adequately investigated. Given that this parameter influences the model's performance, it is crucial to explore its effect across different datasets. Specifically, an analysis of whether a larger number of components may be beneficial for more complex datasets would provide valuable insights into optimizing the model’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cCQeNb1fwY", "reviewer": "Reviewer_23Gv", "summary": "This paper proposes a feature transformation methodology using concept axes, which are the principal components of the difference vectors between text embeddings of prompts specially designed to ignore nuisance attributes/highlight important attributes for anomaly detection.", "questions": "Please address the points raised in the Weakness section. Also:\n\n1. What is the purpose of including Aux. prompts? \n\n2. How does different CLIP architecture and also different VLMs affect performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The methodology is interesting and a solid contribution to this direction of research in vision-language modelling for anomaly detection.\n\nThe results appear to be promising in the experiments presented, although a wider range of experimental setups would be more convincing (see weakness) \n\nThe ablation study is comprehensive.", "weaknesses": "1. Figure 1 is not particularly intuitive or clear, and it is not explained in the text. \n\n2. As the exact formulation of prompts is absolutely critical for this methodology, it should have more dedicated explanation in the main text of the paper, not relegated almost entirely to the appendix. \n\n3. There are not many baselines, and it would have been more convincing if you compare more baselines with and without LAFT transformations. \n\n4. The range of experiments presented are quite restricted. For example with Coloured MNIST, it appears that only one number-colour combination as the normal set was tried. It would be more proper to conduct multiple experiments with different combinations of attributes and show the average result. The same can be said for the other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2sL4jkyzax", "reviewer": "Reviewer_dr7M", "summary": "On the basis of existing anomaly detection methods based on visual language alignment, this paper proposes using task related languages for task oriented feature information screening and transformation to improve the model's anomaly detection capability. The experiment was conducted on multiple datasets and demonstrated better performance compared with existing methods.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is with clear motivation.\n2. This paper is well-organized and easy to follow.", "weaknesses": "1. The criteria for selecting text prompts are ambiguous. Some datasets utilize the category names of the samples, while others employ diverse descriptions. These approaches rest on the critical assumption that anomalies are distinctly defined, exemplified by MNIST, where anomalies arise from differences in numerals rather than variations in handwriting styles or colors. Should the actual anomalies diverge from these presuppositions, might the proposed model's performance diminish relative to methods devoid of textual guidance? In other words, could the model forfeit its capacity to detect all possible anomalies?\n\n2. In the MVTec dataset experiment, the author opted not to employ the concise anomaly descriptions provided by the dataset itself for text prompts, instead relying solely on item categories, mirroring the approach of WinCLIP. What rationale informed this decision?\n\n3. The proposed model is an extension of WinCLIP, yet it appears to forgo the anomaly segmentation functionality inherent to WinCLIP. Is this omission attributable to certain design elements that potentially diminish the model's anomaly localization capabilities?\n\n4. Experiments have been conducted on synthetic datasets like MNIST and CelebA by altering the original datasets. While I acknowledge the challenge of selecting appropriate text prompts for real-world datasets such as MVTec, the author should endeavor to incorporate more authentic datasets into their study, such as the VisA dataset utilized in WinCLIP or the medical AD benchmark employed in MVFA [a].\n\n[a] Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)