Red PANDA: Disambiguating Image Anomaly Detection by Removing Nuisance Factors
Authors: Niv Cohen
Jonathan Kahana
Yedid Hoshen
Venue: ICLR 2023 poster (2023)
Keywords: Anomaly Detection
Disentanglement

Abstract: Anomaly detection methods strive to discover patterns that differ from the norm in a meaningful way. This goal is ambiguous as different human operators may find different attributes meaningful. An image differing from the norm by an attribute such as pose may be considered anomalous by some operators while others may consider the attribute irrelevant. Breaking from previous research, we present a new anomaly detection method that allows operators to exclude an attribute when detecting anomalies. Our approach aims to learn representations which do not contain information regarding such nuisance attributes. Anomaly scoring is performed using a density-based approach. Importantly, our approach does not require specifying the attributes where anomalies could appear, which is typically impossible in anomaly detection, but only attributes to ignore. An empirical investigation is presented verifying the effectiveness of our approach.

Published as a conference paper at ICLR 2023
RED PANDA: DISAMBIGUATING IMAGE ANOMALY DE- TECTION BY REMOVING NUISANCE FACTORS
Niv Cohen Jonathan Kahana Yedid Hoshen
School of Computer Science and Engineering The Hebrew University of Jerusalem, Israel
n vvvccss.. huji i. aac.
ABSTRACT
Anomaly detection methods strive to discover patterns that differ from the norm in a meaningful way. This goal is ambiguous as different human operators may find different attributes meaningful. An image differing from the norm by an attribute such as pose may be considered anomalous by some operators while others may consider the attribute irrelevant. Breaking from previous research, we present a new anomaly detection method that allows operators to exclude an attribute when detecting anomalies. Our approach aims learn representations which do not contain information regarding such nuisance attributes. Anomaly scoring performed using density-based approach. Importantly, our approach does not require specifying the attributes where anomalies could appear. which is typically impossible anomaly detection, but only attributes ignore. An empirical investigation is presented verifying the effectiveness of our approact
INTRODUCTION
Anomaly detection, discovering unusual patterns in data, is key capability for many machine learning and computer vision applications. In the typical setting, the learner is provided with training data consisting only of normal samples, and is then tasked with classifying new samples as normal or anomalous. It has emerged that the representations used to describe data are key for anomaly detection in images and videos (Reiss et al. 202T). Advances in deep representation learning (Huh al. 2016) have been used to significantly boost anomaly detection performance on standard benchmarks. However, these methods have not specifically addressed biases in the used data. Anomaly detection methods which suffer from the existence of such biases may produce more overall errors, and incorrectly classify as anomalies some types of samples more than others. A major source for such biases is the presence of additional, nuisance factors (Lee & Wang 2020).
One of the most important and unsolved challenges of anomaly detection is resolving the ambiguity between relevant and nuisance attributes. As motivating example let consider the application of detecting unusual vehicles using road cameras. Normal samples consist of images of known vehicle types. When aiming to detect anomalies, we may encounter two kinds of difficulties: (i) The distribution unknown vehicles (anomalies) is not known at training time. E.g.. unexpected traffic may come in many forms: horse cart, heavy construction equipment, even wild animals. This is the standard problem addressed by most anomaly detection methods (Ruff et al.) [2018) Reiss et al. Tack et al. 2020). (ii) The normal data may be biased. For example, assume all agricultural machinery appearing during the collection of normal data was moved towards the farmlands. During inference performed on another season, we may see the same equipment moving the other side (and from different angle). This novel view might be incorrectly perceived as an anomaly.
2021; Tack
Unlike previous works, we aim disambiguate between true anomalies (e.g., unseen vehicle types) and unusual variations of nuisance attributes in normal data (e.g., a known vehicle observed previously only in another direction). Detecting normal but unusual variations according to nuisance attributes as anomalies may be source of false positive alarms. In addition, they may introduce undesirable imbalance in the detected anomalies, or even discriminate against certain groups. There are many
'The presented benchmarks are available on github under; hatp://gthub.cmmiiiCccclAAAAAA

---

Published as a conference paper at ICLR 2023
settings where some attribute combinations are missing from the training dataset but are considered normal: assembly line training images may be biased in terms of lighting conditions or camera angles while these may be irrelevant their anomaly score; photos of people may be biased in terms of ethnicity, for example when collected in specific geographical areas. Moreover, in some cases, normal attribute combinations may be absent just due to the rarity of some attributes (e.g. rare colors with specific car models).
The task of learning ignore nuisance attributes requires a general approach. While simple heuristics might sometimes be possible, they suffer from inherent weaknesses: (i) lack of generalization to new image types and nuisance attributes (ii) targeting a specific type of anomalies, which means they will fail to generalize to new, unexpected anomalies. While nuisance attribute removal is easy when the representation is already disentangled in nuisance and relevant components (e.g., some tabular data settings), most image representations and highly entangled.
Our technical approach proposes to ignore nuisance attributes by learning representations that are independent from them. Our approach takes as input training set of normal samples with labeled nuisance attribute. We utilize a -opain--uperrvsed disentanglement approach (Kahana & Hoshen 2022) to remove the information associated with the provided nuisance attribute, while preserving as much uncorrelated information as possible about the image. Specifically, we train an encoder with additional per-domain contrastive loss term to learn representation which is independent of the labeled nuisance attribute. For example, an encoder guided to be invariant to the viewing angle would be trained to contrast images of cars driving the left with similar images, but not against images of cars driving to the right. Additionally, conditional generator is trained on the representations with a reconstruction term, to ensure the representations are informative. We stress that we only use the reconstruction loss to encourage the informativeness of our encoder, and do not use the reconstruction errors to score anomalies. The combination of the two loss terms yields informative representations which are less sensitive to the nuisance attributes. Although our obtained representation is far from being completely invariant to the nuisance attributes, it provides significant gains on several benchmarks. The representations are then combined with standard density estimation methods (k nearest neighbors) for anomaly scoring.
Our setting differs from previous ones, it only relies on nuisance attribute labels. Few anomaly detection algorithms consider the case where the training set contains attribute labels and therefore most methods do not aim to ignore nuisance attributes. normal data are labelled with the value of the relevant attribute and that anomalies belong to a novel class, outside the set of labelled values (Salehi et al. 2021 Hendrycks et al. 2020 Hendryeks & Gimpel 2016). The weakly supervised setting assumes future anomalies will be similar to a few labelled anomalous samples available during training (Cozzolino al. 2018 Gianchandani et al. 2019; Deecke et al.) (2021). However, this type of knowledge is often limiting due to the inherent unpredictability of anomalies. In contrast, we only require knowledge of the factors that are not indicative of the anomalies we wish to find while assuming no specific knowledge of the expected anomalies. In fact, labels for attributes we wish to ignore are often provided by the datasets, such as information about the sensor used to collect the data. In other cases, such labels are easily predicted using pre-trained classifiers such as CLIP (Radford et al. 2021).
detection assumes that
As this task is novel, we present new benchmarks and new metrics for evaluation. Our benchmarks incorporate normal examples which experience unusual variation in a nuisance attribute. Our evaluation metrics measure both the overall anomaly detection accuracy, as well as the false alarm rate due to mistaking normal samples with nuisance variation as anomalies. Our experiments indicate that using our approach for removing the dependencies improves these metrics on our evaluation datasets. While our method can currently handle only quite simple cases, this study indicates way forward for tackling more realistic cases.
nuisance attribute from the representation
Contributions: (i) Introducing the novel setting of Negative Auribute Guided Anomaly Detection (NAGAD). (ii) Presenting new evaluation benchmarks and metrics for the NAGAD setting (iii) Proposing new approach, REpresentation Disentanglement for Pre-trained Anomaly Detection Adaptation (Red PANDA), using domain supaiis-uperviied disentanglement address this setting. (iv) Demonstrating the potential of our approach through empirical evaluation.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORKS
Classical anomaly detection methods. These may be grouped into three themes (Yang et al. 2021 1
Ruff et al, 202T): (i) Density-estimation based methods. Estimation of the density of the normal data can be non-paraeetric methods, such as kNN or kernel density estimation. Parametric methods, such as Gaussian Mixture Models (GMM) (Li et al. 2016) learn a parametric representation of the data to estimate the probability density of the test samples. (ii) Reconstruction- -basiioon-aaedddd methods methods such as PCA learn to reconstruct well normal training samples. Anomalies coming from a different distribution might not reconstruct as well. (iii) One class classification methods A classification approach separating the normal samples and the rest of feature space (e.g. SVDD (Tax & Duin )004))
Deep anomaly detection methods. As only normal samples are available during training, we cannot learn features with standard supervision. Therefore, deep anomaly detection methods either use self-supervision learning to score the anomalies (Hendrycks et al., 2019). or adapt a pre-trained
representation (Hendrycks et al. 2019; Reiss et al.] [202]] Reiss & Hoshen 2021 Ruff al.. 2018)
Perera & Patel (019) to describe the normal training data. (i) Self-supervised methods these methods
learn to solve an auxiliary task on the normal samples, test the performance on new images, and score anomalies accordingly: the network is expected to perform better on the normal samples that come from a similar distribution (Hendrycks et al. 2019). More recent works such as CSI (Tack et al. (020) or DROC (Goyal et al., (2220) use contrastive learning to learn representation of the normal data. (ii) Adaptation of Pre-trained Feature- Transfer learning pre-trained features was shown to give strong results for -fiittiibbtttii detection by (Hendryeks et al.) 2019). Adaptation of pre-trained features for anomaly detection was attempted by Deep-SVDD (Ruff et al., 2018). which adapted features learnt by an auto-encoder using compactness loss. Perera & Patel suggested training the compactness loss jointly with ImageNet classification (Perera & Patel 2019). By incorporating early stopping and EWC regularization (Kirkpatrick et al. 2017). PANDA (Reiss al.. 2021) allowed feature adaptation with mitigated catastrophic forgetting, resulting in better performance. Further improvement in pre-trained feature adaptation was later suggested by MeanShifted (Reiss & Hoshen 2021] using contrastive learning to adapt the pre-trained features to the normal training set.
Domain-supervisedd disentanglement. Disentanglement is the process of recovering the latent factors that are responsible for the variation between samples in a given dataset. For example. from images of human faces we may recover the age of each person, their hair color, eye color, etc. In domain-supervised disentanglement, one assumes that a single such factor is labelled and aims to learn a representation of the other attributes independent of the labelled factor. This task was approached with variational auto-encoders (Jha et al. 2018 Bouchacourt et al. 2018 and latent optimization (Gabbay & Hoshen| 2019 2021). Contrastive methods have also shown great promise with general disentanglement (Zimmermann et al. (2021). This was followed by Kahana & Hoshen in domain- supervised disentanglemen (Kahana & Hoshen 2022) who employed a contrastive loss for each samples individually, learning a code which ideally describes only (and all) attributes which are uncorrelated to the labelled attributes. Domain-supervised disentanglement has been used for a variety of applications. Most notably, for generative models (Zhu et al. 2011 2 (Gabbay & Hoshen 2019). Disentangiement models have also been discussed in the context of interpretability (Hsu et al. 2017). abstract reasoning (Van Steenkiste et al., 2019), domain adaptation (Peng et al.. 2019), and faimess (Creager et al., (019). Some previous works have considered using domain supervision for increasing fairness in anomaly detection (Davidson & Ravi 2020; Zhang & Davidson 2021 Shhkkhar et al.] 2021). }. These methods aim at obtaining equal anomaly detection performance across the protected attributes. On the other hand, our objective is to ignore the nuisance attributes in order to improve the overall performance of the anomaly detection method.
set of similarly-
NUISANCE ATTRIBUTES MISLEAD ANOMALY DETECTORS
Anomaly detection methods aim to detect samples deviating from the norm. However, operators of anomaly detection methods expect the deviation to be semantically relevant. As the anomaly detection setting is typically unsupervised and the type of anomalies cannot be expected, it is impossible to predict in which modes of variations anomalies will appear. Yet, may know in advance that we do not wish to detect anomalies in nuisance attributes. For example, we may wish to detect anomalous vehicles in traffic. Anomalies can appear in many attributes such as car type, color,
3

---

Published as a conference paper at ICLR 2023
condition, headlights type, etc. However, if we do not wish to detect anomalies in the car pose, avoiding false positives associated with this attribute may be possible without knowing in advance in which attributes anomalies will appear.
Current algorithms rely on different inductive biases to select the relevant attributes and remove the nuisance ones. The most common choice is manual feature selection, where the operator specifies particular features that would be the most relevant (Pevný) 2016 Gu et al.] 2019). Contrastive learning methods specify augmentations which remove specific attributes (minor color and location variations) from the representation. This helps to select attributes more relevant to object-centric tasks. Similarly, representations pre-trained on supervised object classification (e.g. ImageNet (Deng et al. (2009)). which have recently demonstrated very strong results on image anomaly detection, focus on object-centric attributes the expense of other low-level image attributes. The most extreme level of supervision is the out-of-distribution detection setting, where the class labels are provided for all normal training data, and anomalies are expected to belong to an unseen class. However, this guidance is not available in the typical anomaly detection setting as anomalies are unexpected.
Our novel setting, Negative Attribute Guided Anomaly Detection (NAGAD), allows the specification of nuisance attributes that should be ignored by the anomaly detector. Unlike specifying the relevant attributes, which is not possible in anomaly detection, specifying nuisance attributes is often possible. Users may know in advance about the attributes they wish to exclude for anomaly detection; either due to legal and moral reasons, or due to prior domain knowledge. The issue of excluding such attributes from images remains major technical challenge even when such attributes are known.
natural way for specifying nuisance attributes is to provide labels for them. For example, wishing detect anomalies according to car model but according to its pose, may provide for each image a label for the car pose (see Fig D Currently available anomaly detection approaches cannot directly benefit from such information and thus mitigate nuisance attributes only implicitly (using the mechanisms explained above). In Sec. |
we describe a specific technical approach for using the guidance for anomaly detection. Yet, we stress that our main contribution the novel anomaly detection setting.
4 RED PANDA: DISENTANGLEMMET APPROACH FOR REMOVING A NUISANCE FACTOR
We detail the different stages of our approach below. An algorithm box summarizing the different steps can be found in App
4.1 OBTAINING LABELS FOR THE NUISANCE ATTRIBUTE
Our approach, REpresentation Disentanglement for Pre-trained Anomaly Detection Adaptation (Red PANDA), aims achieve a representation invariant nuisance attribute of our dataset, leading to better detection of anomalies expressed in relevant attributes. To do so, we provide labels for the nuisance attribute. For example, when we wish to detect anomalies in driver behaviour, we may wish to ignore the vehicle's 's pose. We can achieve this by providing pose labels during training, and using them to be less sensitive to this attribute.
To achieve these labels, we have few options. In some cases, they may already exist in the dataset. very natural such case is when we have data from a few static cameras, and we wish to ignore the camera identity. In many other cases, a pre-trained classifier, already trained for these specific attributes may provide such labels. Recently, pre-trained models for text-based zero-shot classification such as CLIP (Radford et al. 2021) have shown promising results. Such models allow supplying of-the-shelf labels for a very large set of attributes. We conducted a small experiment over the Edges2Shoes (Isola al.) 2017 7) dataset, automatically labelling with CLIP, and achieved 99.97% accuracy in labelling whether an image is a photo or a sketch (Fig Taken together in many cases labels for nuisance attributes can be achieved virtually no cost.
4

---

Published as a conference paper at ICLR 2023
4.2 PRELIMINARIES
In our setting, the training set denoted as Dirain consists of normal samples only. For each normal image 2; E Dirain we are also provided with label ni describing the nuisance attribute we wish to ignore (the setting can be naturally extended to many nuisance attributes). Our evaluation set Dtest consists of both normal and anomalous samples. We denote the normal/annoall label for a test image 24 y.. For each such dataset, each sample is described by multiple attributes (n4, ai bj. ...) E N x A x B describe different relevant attributes (consider for example the identity of the object, the lightning condition, and camera angles as different attributes). We only assume labels for N during training. We assume that the anomaly label is always function of (potentially) all the relevant attributes = (a bi, Ci. ..).... Namely, we assume the nuisance attribute n, never affects the anomaly label y:- We emphasize that in our described setting, none of the relevant attribute labels nor the anomaly labels are given during training.
X x... where N describe our nuisance attribute, and A, B, C....
We aim to learn an encoder function f mapping samples 24 a code describing their relevant attributes fxx)) E Rd. We also wish our codes to be invariant. This is, we wish our encoder to represent the relevant attributes in way that is not affected by the nuisance attributes:
p(n.) p(n)ff(x))
We also wish our code be informative- to represent sufficient information regarding our relevant attributes (T(;) ) is the mutual information between its two arguments):
I(aa by, c. xx)).))))) = I((a, b,. C....); fxx))
(2)
In practice, the invariance may be measured by the accuracy of predicting from the latent code fz)). Similarly, we can measure the accuracy of predicting the relevant attribute used to define anomalies (informativeness). Empirical evaluations of these measures for datasets can be found in the ! Given such a representation we may later score anomalies independently from any biases caused by the nuisance attribute wish to ignore.
4.3 CONTRASTIVE DISENTANGLEMENT
In this section, we describe the technical approach we employ for ensuring that f does not contain information the nuisance attribute, while retaining much information about the relevant attributes (Kahana & Hoshen 2022).
Pre-trained encoder. We initialize the encoder function f with an ImageNet pre-trained network. ImageNet-pre-trainee trained representations were previously shown to be very effective for image anomaly detection (Reiss et al., 202T). Off-the-sell pre-trained representation, however, also encodes much information on the nuisance attributes. Therefore they do not satisfy our invariance objective.
Contrastive loss. Our objective is that images that have similar relevant attributes but different nuisance attributes would have similar representations. Although we are not provided with supervised matching pairs,
the proxy objective requiring the distribution of representations of images having different nuisance attributes to be the same. To match the distributions we first split our training data Drrain to disjoint subsets Sn, according to the nuisance attribute values:
D Drrain =U+ Sm. n.EN
We then use contrastive loss on each of the sets Sn, independently (sim denotes cosine similarity. Te, Fy E Derein are arbitrary normal samples, and n(x;), (2,,) are their nuisance labels):
Lcon log Σ In(o))=n(ss)) sim sn,5;
(4)
5

---

Published as a conference paper at ICLR 2023
This objective encourages the encoder to map the image distribution of each nuisance attribute uniformly to the unit sphere (Wang& & Isola 2020). Therefore, matches the distributions of sample embeddings coming from difterent values of the nuisance attribute (as required by Eq (Wang & Isola 2020). We note that matching of marginal distributions is necessary, but not a sufficient condition for alignment (Kahanaa & Hoshen 2022
Another problem that may arise is insufficient informativeness: the contrastive objective does not prevent ignoring some of the relevant attributes (Chen et al. [022T) To support the informativeness we add an augmentation loss, encouraging different augmentations of the same image to be mapped to similar codes: Laug -sim (A),
The used augmentations are detailed in appendix D To further encourage informativeness, also employ reconstruction loss.
Reconstruction loss. To encourage the representation to contain as much information about the relevant attributes possible, we reconstruction constraint. Specifically, we require that given the combination of the representation fi (which ideally ignores the nuisance attribute) and the value of the nuisance attribute ni, should be possible perfectly recover the sample z;z This is enforced using generator function G which is trained end-to-end together with the encoder. The reconstruction is measured using perceptual loss.
Lree
Σ
pperceptual (xi, ((fxx)) n.))
Ti
(5)
Derain N
4.4 DENSITY BASED ANOMALY SCORING
Similarly to other anomaly detection methods, we hypothesize the anomalous samples will be mapped to low-density regions, while normal data will be mapped to high-density regions. When the representation contains only relevant attributes, low-density regions would indeed correspond to samples with rare relevant attributes which are likely to be anomalies. To numerically estimate the density of the normal data around each test sample, we use the & nearest neighbors algorithm (kNN). We begin with extracting the representation for each normal sample: fi f(x;). Va, E Dtrain- Next, for each test sample we infer its latent code ft f(xe). Finally, we score by the kNN distance to the normal data:
S(x.)=
Σ sim fe)
f.EN((f.)
where (ft) denotes the most similar relevant attribute feature vectors in the normal data (com- parison of different density estimation methods and different values of l: can be found in App[K). We note that although we trained our encoder f with a contrastive loss, encouraging uniform distribution in the sphere, the high dimension of the latent space allows us to distinguish between high and low-density areas of the distribution of normal data. Runtime considerations are discussed in App
5 EXPERIMENTS
5.1 SETTING
Benchmark construction. As our anomaly detection setting is novel, new benchmarks need to be designed for its evaluation. The following protocol is proposed for creating the benchmarks. First, we select an existing dataset containing multiple labelled attributes. We designate one of its attributes as a nuisance attribute, (e.g., the object pose) and other attributes as relevant (e.g., the identity of the object). Only the relevant attributes are used to designate an object anomalous whereas the nuisance attribute does not. We then remove images featuring certain combinations nuisance and relevant attributes from the training set, creating bias in the data. For example, may remove all left-facing cars for one car model, and right-facing cars for amother car model. As these combinations removed from the normal train set, refer to them that shares all the attributes (including nuisance attributes) with a normal training sample as a familiar samples. In this setting, aim both to both detect true anomalies (anomalies according to the
heudo-anomalies. We refer to any sample

---

Published as a conference paper at ICLR 2023
relevant attributes), and treat pseudo-anomalies normal as the familiar samples, they differ only in nuisance attributes.
Metrics. We wish not only to measure overall anomaly detection performance but also to evaluate the false alarm rate due to pscudo-anomalies We therefore report our results in terms of three different scores. Each uses two subsets of the test set and measures how well our anomaly detection algorithm discriminates between them in terms of ROC-AUC: (i) Standard anomaly detection (AD)-Score, which measures how accurately anomalies are detected with respect the normal test data (both familiar samples and pseudo-anomalies), (ii) Pseudo anomalies (PA)-Score: measures how much pseudo anomalless are scored as anomalous compared familiar samples (iii) Relative abnormality (cA)-score: measures how accurately true anomalies are detected compared to ppeudo-anomalics
5.2 RESULTS
Compared Methods. We compared to the following methods: DN2 (Reiss et al. 2021). MeanShifted (Reiss & Hoshen 2021), CSI (Tack et al.. 2020), SimCIR (Chen et al. 2020). A full description of the compared methods can be found in App.CC.
Datasets. We report the results on three multi-attribute datasets based on Cars3D, SmallNORB and Edges2Shoes. We chose these specific datasets as they are the common datasets in the field of domain-supervised disentanglement (Gabbay & Hoshen 2019: Kahana & Hoshen, 2022). We also find these datasets to be non-trivial for state-of-the-arr anomaly detection algorithms. Full details on each dataset can be found in App E
Cars3D (Reed et al. 2015). A synthetic dataset, where each image is formed using two attributes: model and pose. Car models are varied across different colors, shapes and, functionalities. Each car model is observed from multiple camera angles (pose). To simulate pseudo anomalies, we model and labeled it pseudo-anomaly An illustration of the dataset can be seen in Fig. [ We can see in Tab that the disentanglement approach significantly outperforms methods that do not use any guidance to remove the nuisance attribute. We detect true anomalies, without assigning high anomaly scores to the peeudo-anomalie significantly better than all other methods compared. The RA-Score shows that we distinguish well between true anomalies and pseudo-anomalies.
randomized for each pose single
SmallNorb (LeCun et al. 2004). In this dataset as well we define our nuisance attribute to the camera angles, and pseudo pseudo-anomalee anomalies are car models seen from never-seen-before angle during test time. We can see in Tab that our approach outperforms on this dataset too. All methods utilizing pre-trained features detect true anomalies fairly well. This is expected, as the network learns good representation of objects during pre-training. Our disentanglement approach significantly reduces the tendency to score secudo-anomalic as anomalies. CSI treats pseudo- anomalies similarly to normal samples, but this is most likely because its representation for this dataset is not informative, and
Type-1
Type-2
Type-3
Type-4
Type-5 (Anom.)
Pose 1
Pose 4
Pose
6
Pose 17
Figure 1: Samples from the Cars3D datasets. anommalss are marked in green while true anomalies are marked in red. Both pseudo-anomalies and true anomalies appear only in the test set.
7

---

Published as a conference paper at ICLR 2023
Table 1: Empirical Evaluation the Cars3D Dataset (ROC-AUC)
Dataset
Method
AD-Score (ft)
PA-Score (1)
RA-Score (1))
SimCLR
0.780
0.519
0.741
CSI
0.606
0.579
0.538
DN2
0.946
0.564
0.916
Cars3D
MeanShifted
0.943
0.595
0.917
Ours
0.985
0.506
0.980
does not distinguish well between unseen data (true anomalies or pseudo-anomalies) and the familiar samples.
Table 2: Empirical Evaluation on the SmallNorb Dataset (ROC-AUC)
Dataset
Method
AD-Score (ft)
PA-Score (1)
RA-Score (f)
SimCLR
0.805
0.728
0.638
CSI DN2
0.618
0.556
0.575
0.908
0,819
0.768
SmalINorb
MeanShifted
0.948
0.870
0.811
Ours
0.953
0.581
0.943
Edges2Shoes (Isola et al. 2017. This dataset contains photos of shoes and edge maps images of the same photos. An illustration of the dataset can be seen in Fig. 2 This dataset is challenging as the photo and sketch domains are quite far, making the nuisance attribute dominant. E.g., by observing only sketches of boots, real photos of boots could be easily considered as anomalies without further guidance. Our approach outperforms methods that do not remove nuisance attributes from the representation. scored higher than normal images by our approach, their scores still lower than the true anomalies (demonstrated by the RA-score). Our approach significantly outperforms the baselines, showcasing the importance of specifying and removing nuisance attributes.
observe (by the PA-score) that although the pseudo-anomalies are indeed
Table 3: Empirical Evaluation on the Edges2Shoes Dataset (ROC-AUC)
Dataset
Method
AD-Score (1t)
PA-Score (1)
RA-Score (1)
SimCLR
0.567
0.642
0.510
CSI
0.574
0.873
0.412
DN2
0.500
0.631
0.455
Edges2Shoes
MeanShifted
0.486
0.790
0.386
Ours
0.781
0.711
0.719
In summary, while some methods outperformed our method in terms of PA score on some experiments, this done by scoring both pseudo-anomalies and true anomalies normal samples (resulting in significantly worse AD-Score). The PA score alone can simply be optimized by a random anomaly detector getting ROC-AUC of 0.5. The RA-score measures the ability to distinguish true anomalies from pseudo anomalies directly. We significantly outperform all baselines on this score.
DISCUSSION & LIMITATIONS
Multi-attribute dataset. Many datasets (e.g. SmalINorb) have more than two attributes. In some cases, we may wish remove multiple nuisance attributes. Methods such as (Gabbay & Hoshen 2019) very naturally extend to the case of disentangling many factors the same dataset. These methods operate using
bottlenecks. Multiple attributes be excluded by concatenating their representation to the latent code (Gabbay al. 2021). Our approach can be extended to the case of multiple attributes using such methods.
8

---

Published as a conference paper at ICLR 2023
Applying approach to other data modalities. While this work is focused image datasets, disentanglement approaches may assist anomaly detection efforts in other modalities as well. This includes modalities such as audio signals (Abeßeer & Müller 2021 or text (Cheng et al. 2020), and also scientific data such as Single-Cell data (Hetzel et al.).
Table 4: Empirical Evaluation on the MVTec-AD Dataset With Nuisance Attributes (ROC-AUC)
PatchCore
No Nuis.
JPEG
JPEG+
Contr.
Contr.+
Gauss.
Gauss.+
AD-Score (1)
0.991
0.848
0.860
0.914
0,915
0.895
0,900
PA-Score (1)
0.977
0.976
0.980
0.980
0.972
0.969
RA-Score (1))
0.701
0.725
0.833
0.836
0.796
0.806
Domain supervised disentanglement in the wild. Currently, state-of-the-ar domain-supervise disentangiement methods achieve impressive results on synthetic or curated datasets. Such methods do not perform as well for in-the-wild datasets. As our approach heavily relies on disentanglement, is prone to similar limitations. As the field of disentanglement advances, the advancements can be translated to improved anomaly detection capabilities using our approach. To evaluate the progress of future anomaly detection methods, we also include harder benchmark (Tab(6). We adapt standard anomaly detection benchmark (MVTec-AD, (Bergmann et al.) 2019)). and evaluate state- of-the-art method (PatchCore (Roth al. (02I)) that achieves very high accuracy the original data ("No Nuis."). However, in the presence of nuisance factors, that create pseudo-anomalies in the test set (JPEG compression, Contrast Augmentation, Gaussian noise), the anomaly detection results significantly deteriorate. The method we proposed cannot apply to this dataset for multiple reasons: (i) it is designed for coarse-grained rather than fine-grained AD (ii) struggles with complex real-world images. Extending the state-of-the- to such datasets is an open challenge. Additional details about this benchmark can be found in App A
is not solved by the methods explored in this paper (including ours), and is left for future research.
Highly biased datasets. Similarly other disentanglement approaches, require the distributions of relevant attributes across nuisance domains to be somewhat similar. We have shown that our method can work when the supports across domains not overlapping. Still, we expect that when the supports are highly coneolallap the results will significantly deteriorate. Developing methods able to disentangle domains with highly non-overlapping support is an exciting future direction.
Imperfect invariance. While our method aims to achieve invariance, the results are still imperfect. We report the invariance and informativeness of our learned representations Tab Although far from optimal, the method already provides significant gains for the anomaly detection task (Sec
Missing and mislabeled nuisance factors. As our approach relies on labeled nuisance factors, might be sensitive to mislabeled nuisance labels. Although wrong labels can hinder the efforts of most machine learning algorithms, our approach might be sensitive to two other types of errors: (i) our technical methods rely on categorical labels for the nuisance attribute. In experiments successfully address this by quantization of the continuous variable. Yet, this procedure should be carefully examined for each application. (ii) if a user fails to identify the correct nuisance factor, our method could not be used to remove it.
Further Discussion. Further discussion regarding Supervised self-supervised pre- pre-training Supervised vs. self-supervised pret-raining Removing nuisance attributes with generative models
7 CONCLUSION
We proposed a new anomaly detection setting where information is provided on a set of attributes that are known to be irrelevant for distinguishing between normal samples and anomalies. Using a disentanglementtbean
approach, we showed how this additional supervision can be leveraged for better anomaly detection in biased datasets. As identifying irrelevant attributes is easier than predicting in which attributes anomalies will appear, we expect further research on this new setting to be fruitful and promising.

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGEMENT
This research was partially supported by the Israeli Science Foundation, the Hebrew University Data Science grants (CIDR), and the Israeli Council for Higher Education.
REFERENCES
Jakob Abeßer and Meinard Müller. Towards audio domain adaptation for acoustic scene classification
using disentanglement learning. arXiv preprint arXiv:2110.13586, 2021.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehensive world dataset for unsupervised anomaly detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9592-9600, 2019.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng. and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.
Yunqiang Chen, Xiang Sean Zhou, and Thomas S Huang. One-class for learning in image re- trieval. In Proceedings 200 international conference on image processing (Cat. No. 01C372005) volume 1, pp. 34 37. IEEE, 2001.
Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. Improving disentangled text representation learning with information- theoretic guidance. arXiv preprint arXiv:2006.0006 006993 2020.
Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint T.Xi::2005.02557. 2020.
Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias NieBner, and Luisa Verdoliva. Forensictransfer: Weakly-supervised domain adaptation for forgery detection. arXiv preprint arXiv: 025100, 2018.
Elliot Creager, David Madras, Jömn-Hennn Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement, In International conference on machine learning, pp. 1436-1445. PMLR, 2019.
Ian Davidson and Selvan Suntiha Ravi. A framework for determining the fairness outlier detection. In ECAI 2020, pp. 2465-2472. IOS Press, 2020.
Lueas Deecke, Lukas Ruff, Robert A Vandermeulen, and Hakan Bilen. Transfer-based semantic anomaly detection. In International Conference on Machine Learning, pp. 2546-2558. PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. arXiv preprint 1996v //796, 2019.
10

---

Published as a conference paper at ICLR 2023
Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7783,6792, 2021.
Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An imgge is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems, 34: 9216-9228, 2021.
Urvi Gianchandani, Praveen Tirupattur, and Mubarak Shah. Weakly-supervised spatiotemporal anomaly detection. University of Central Florida Center for Research in Computer Vision REU, 2019.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. Droce: Deep robust one-class classification. In International Conference on Machine Learning, pp- 3711-3721. PMLR, 2020.
Xiaoyi Gu, Leman Akoglu, and Alessandro Rinaldo. Statistical analysis of nearest neighbor methods for anomaly detection. Advances in Neural Information Processing Systems, 32, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXhv:1610.02136, 2016.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self seff-superviee learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems, 32, 2019.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve -uttof-distribution robustness. arXiv preprint 20X1v:2004.06//00 2020.
Leon Hetzel, Simon Boehm, Niki Kilbertus, Stephan Günnemann, Mohammad Lotfollahi, and Fabian Theis. Predicting cellular responses to novel drug perturbations at a single-cell resolution. In Advances Neural Information Prucessing Systems.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems, 30, 2017.
Minyoung Huh. Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv:/608 1608.08614 14, 2016.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VSR Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In Proceedings of the European Conference Computer Vision (ECCV), pp. 805-820, 2018.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Jonathan Kahana and Yedid Hoshen. A contrastive objective for learning disentangled representations. arXiv preprint arXiv:2203: /284, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
11

---

Published as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Grabska-Barwinsker Barwinska, et al. Overcoming
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, 2:11-104 Vol.2, 2004.
Wei-Yu Lee and Yu-Chiang Frank Wang. Learning disentangled feature representations for anomaly detection. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 2156-2160 IEEE, 2020.
Lishuai Li, R John Hansman, Rafael Palacios, and Roy Welsch. Anomaly detection via a gaussian mixture model for flight operation and safety monitoring. Transportation Research Part C: Emerging Technologies, 64:45-57, 2016.
Xingchao Peng, Zijun Huang. Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled representations. International Conference on Machine Leurning, pp. 5102-5112. PMLR, 2019.
Pramuditha Perera and Vishal M Patel. Learning deep features for one-class classification. IEEE Transactions on Image Processing, 28(11)5540--5433 2019.
Tomáš Pevný. Loda: Lightweight on-line detector of anomalies. Machine Learning, 102(2):275-304, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8748-8763. PMLR, 2021.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. Advances in neural information processing systems, 28, 2015.
Tal Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. arXiv preprint arXiv:2106.03844 038444 2021.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2806-2814, 2021.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. arXiv preprint arXhv:2106.0822 2021.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393 4402. PMLR, 2018.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-795, 2021.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and distr--iittiitttion detection: Solutions and future challenges. arXiv preprint
2021.
Shubhranshu Shekhar, Neil Shah, and Leman Akoglu. Fairod: Fairness-aware outlier detection. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 210-220, 2021.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Novelty detection via contrastive
12

---

Published as a conference paper at ICLR 2023
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54(1):45-66, 2004.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? Advances Neural Information Processing Systems, 32, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939. PMLR, 2020.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22863-12877 2021.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.1133: 2021.
Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2014.
Hongjing Zhang and lan Davidson. Towards fair deep anomaly detection. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 138 138-148, 2021.
Yun Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. Advances in neural information processing systems, 31, 2018.
Roland Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
A MVTEC-AD BASED ANOMALY DETECTION WITH NUISANCE FACTORS
We wish to evaluate the NAGAD setting on more realistic benchmark. Therefore, we adapt the MVTec-AD benchmark include nuisance factors. We extend the pseudo-anomalies: normal test samples that underwent an augmentation simulating different image source (see Sec[A.I). We also include the true anomalies twice: once in their original form, and once with the augmentation. We down-sample the true anomalies by a factor of 5 to keep the class sizes relatively balanced. can be seen in Tab[A including pseudo-anomalies significantly deteriorates the anomaly detection capabilities with respect to the original data. The -noomaless differ from the normal data (Tab[A] PA-score) and are often considered more anomalous than the true anomalies (Tab A RA-Score).
of each class include
To allow future methods the ability to adapt, and more invariant to the nuisance variation, we also include version with extended dataset in our benchmark (Tab[A] JPEG+; Contr.+; Gauss.+). In this version the normal training data include both the original and augmented normal data, from two unrelated MV'Te classes ("transistor" & "zipper"). Using the labels the nuisance variation for these classes, we expect future methods to he able to pay less attention to the nuisance attribute. We therefore do not include the MVTec-AD classes "transistor" & "zipper" in our average image-level ROC-AUC. As the other discussed methods perform less well also on the original MVTec-AD dataset, we not include their results.
A.1 AUGMENTATIONS AS NUISANCE ATTRIBUTES
To simulate the nuisance attribute, we consider three augmentations, chosen from the augmentations used by Hendrycks & Dietterich (2019). For each chosen augmentation we specify its severity degree defined there. We consider the Tollowing augmentations:
-PEGJJPEC compression artifacts taken by severity degree 5, compressing each image to 7% of its original size.
Contrast Decrease of contrast for each image (severity degree 3).
Gaussian Noise Blurring each image (severity degree 5). Specifically, the blurring gaussian kernel has a standard deviation of 3.
We illustrate the chosen augmentations on one image in Fig A.I
B FURTHER DISCUSSION
Supervised vs. self-supervised pre-training, Many top-performing approaches (including ours) rely eeoemally-pretrainee weights for initializing their neural networks. Pre-trained weights implicitly provide useful guidance regarding the relevant attributes we should focus on, and the ones may wish to ignore (e.g. low-level image information). Different pre-trained networks provide different relevant/nuisance attribute splits. We found that pre-trained weights obtained from supervised classification external datasets such ImageNet, tend to emphasize the main object featured in the center of the image, and are more invariant to other attributes. Representations
Original
Gaussian Noise
Contrastive
JPEG
A glossary of the augmentations used to create the nuisance factors in Tab An original image, Gaussian noise augmentation, Contrastive augmentation, and JPEG Compression augmentation.
14

---

Published as a conference paper at ICLR 2023
learned by self-supervised pre-training external datasets affected both by the dataset and by the augmentation used for its contrastive learning. Therefore they may have different inductive biases.
Augmentations. Different methods may require augmented images to be similar dissimilar to the original image (Chen al. 2020 Tack et al. 2020). This choice tends to have a strong effect on the results. E.g., network trained to be rotation invariant may fail when the relevant attributes include the image orientation angle. Our approach only uses simple augmentations such as Gaussian blurring, saturation, and crops. We expect these augmentations vast majority of cases. In general, augmentations should be carefully inspected when deploying anomaly detection methods in practice.
restrict the anomalies detectable in the
Removing nuisance attributes with generative models. Recently, generative models e.g. StyleGAN (Karras al.] 2019) have been able learn very powerful representations for several data types. particularly images of faces. Their representations exhibit certain level of disentanglement [Wu al., (02T). When available, such models can be utilized for removing nuisance attributes similar approach to ours.
C COMPARED METHODS
DN2 Reiss al 2027). simple but effective approach fully reliant on pre-training. It uses an SrngeNet-pretrainee network to extract representations for each image. Each test image is scored using ANN density estimation similarly to our approach. MeanShifted (Reiss & Hoshen 2027). A recent method that achieves state-of-the-art performance standard anomaly benchmarks. It uses a modified contrastive learning loss to adapt its feature to the normal train set. This method the same pre-trained network our method initialize the features. It then uses kNN for anomaly scoring. CSI (Tack el al.) 2020). A strong self-supervised anomaly detection method that does rely on pre-training. uses types of augmentations: fine augmentations simulating positive contrastive loss samples, and domain shifts simulating negative samples. Anomaly scoring performed using an ensemble of similarity scores based on the learnt features. SimCLR (Chen er al. 2020). An ablation of our approach that trains single contrastive loss rather than different contrastive loss for each domain. We score the anomalies similarly our approach.
D IMPLEMENTATION DETAILS
D.1 DISENTANGLEMENT MODULE
We use most of the parameters as in the DCoDR paper (Kahana & Hoshen, 2022) for our disentangle- ment module. All images were used in 64 64 resolution. For the contrastive temperature, we use 7 0.1 for all the datasets. We scale down the loss Eree by a factor of 0.3.
Boots
Sandals
Shoes
Slippers (Anom.)
Photos
Sketches
Figure 2: Samples from the Edges2Shoes datasets. Pseudo- anomalls are marked in green while true anomalies are marked red. Both pseudo-anomalies and true anomalies appear only in the test set.
15

---

Published as a conference paper at ICLR 2023
Architecture. We used ResNet50 encoder pre-trained image classifications. In accordance with previous works, we add 3 Cully-connected layers to the encoder for the SmallNorb dataset (Gabbay Hoshen 2019 Kahana & Hoshen 2022). For the perceptual loss of the generator used VGG network pre-trained on ImageNet.
Optimization. We use 200 training epochs. In each batch used 32 images from 4 different nuisance classes (a batch size 128, in total). We used a learning rates of 10 and 3 10 for the encoder and generator (respectively).
Augmentation. We used Gaussian blurring (kernel_size = 5, = 1), high contrast (contrast (T.8,3.0), and high saturation (saturation (1.8, 3.0)) for augmentation. For Edges2Shoes we used only Gaussian Blurring. For the SimCLR (Chen et al., 2020) contrastive learning (both in our approach and the baseline), we follow DCoDR by only augmenting the original image once, and comparing the augmented and the original views encodings. This is in contrast to SimCLR which compares two augmented views instead.
Baselines. We ran all the baselines using the same ResNet50 backbone network we used. As CSI (Tack et al. 2020) performance deteriorated when using ResNet50, we ran it with the original ResNet18 backbone used by the authors.
D.2 HYPERPARAMETER TUNING
As we aim to detect anomalies without relying on any labelled anomalous samples, we used all the baselines with their default parameters. As DCoDR (Kahana Hoshen 2022) different temperature parameter for each dataset, we used т = 0.1 for all datasets.
D.3 SCORING MODULE
use faiss (Johnson et al. 2019) &NN implementation, using 1.
E DATASETS
We label each sample as either normal, true anomaly, or pseudo-anonaly as detailed below. We include only true anomalies and pseudo-anomalies in the test set, and split the normal samples between the training set and the test set (85% /15% train/test split). To simulate anomalies in the dataset, first designate true anomalies described in Sec We then chose combinations of normal classes and the nuisance attribute to designate pseudo anomalies. We used the following random combinations for pseudo anomalies:
Cars3D: We define true anomalies as images of (randomly selected) car models. The nuisance
attributes are defined in Tab
Table S: Cars3D Pseudo Anomalies Selection
Azimuth
Object Type
Azimuth
Object Type
173
12
48
16
13
66
75
14
32
23
15
153
4
44
16
128
78
17
120
6
108
18
38
19
8
167
20
106
182
21
4
10
99
22
175
78
23
111
16

---

Published as a conference paper at ICLR 2023
SmallNorb: Each image is synthetically constructed from several attributes: object type, camera azimuth, camera elevation, and lighting. The object types come from different categories such as animals, people, planes, trucks, and cars. To simulate anomalies we randomized single object class (e.g. deer) from each category type. We define the camera azimuth angles as our nuisance attribute. For each given azimuth angle randomize single object class, and assign samples of that type and camera angle as pseudo-anomalies. The nuisance attributes are defined in Tab
Table 6: Smallnorb Pseudo Anomalies Selection
Azimuth
Object Type
Azimuth
Object Type
44
9
38
17
10
35
2
9
11
12
25
12
24
4
48
13
35
20
14
29
6
12
15
23
7
44
16
41
8
8
17
43
Edges2Shoes: The images in this dataset are labelled in terms of image type (sketch vs. photo), shoe type, and other attributes (we use labels from the UT-Zappos50K dataset (Yu & Grauman 2014)). We assign all images with shoe type 'slippers' true anomaly. We assign all photos of type 'sandal', and all sketches of type 'boot' as pscudo-anomalic The nuisance attributes are defined in.
Table 7: Edges2Shoes Pseudo Anomalies Selection
Image Type
Shoe Type
Photo
Sandals
Sketch
Boots
We included all of the pseudo anomalies the test set.
E.
COMPUTE RESOURCES
The entire project used in total 3000 hours of NVIDIA RTX A5000 GPU (including development, testing, and comparisons). All resources were supplied by local internal cluster.
E.2 RUNTIME
Although &NN has runtime complexity linear in the number training data, it can sped up using K means core-set techniques (as done in SPADE (Cohen Hoshen 2020) PatchCore (Roth Øt al. 2021T). In practice, the wall-clock runtime of the retrieval stage of our approach is minimal, even without such speedups (>3500 images per second for the SmallNorb dataset).
E.3 LICENSE
Our technical approach is based the DCoDR paper(Kahana & Hoshen 2022) with SOFTWARE
RESEARCH LICENSE detailed here" The implementation uses the PyTorch and faiss (Johnson
al.
2019) packages. PyTorch Uses a BSD-style license, detailed in their license file faiss uses
MIT
License.
The CadPoRadiord et al. 202 network we used for automatic labelling uses MIT License.
SimCLR (Chen et al. 2020 used by DcoDR and as baseline uses Apache License.
17

---

Published as a conference paper at ICLR 2023
F REMOVING A RANGE OF ANGLES AS PSEUDO-ANOMALIES
We performed an additional experiment where we select pseudo-anomalies from range of values, instead considering just one viewing angle for each class for the peudo-anomallis. Here, for set of 24 random classes we randomly select range of 36 degrees in azimuth, which we remove from the train set, and treat as pseudo anomalies. report the results for the SmalINorb dataset Tab We can see our method still performs well under this setting.
Table 8: Empirical Evaluation the SmallNorb Dataset With Range of Angles Pseudo- Anomalies (ROC-AUC)
Dataset
Method
AD-Score (f)
PA-Score (1)
RA-Score (1)
SimCLR
0.797
0.743
0.633
DN2
0.908
0.819
0.768
McanShifted
0.949
0.872
0,812
SmalINorb
Ours
0.949
0.557
0.942
G ABLATION STUDY
We further extend our ablation study in Tab 9 We provide versions of our methods, without one or more of its components. (i) First, we validate the need for our use of labels for the nuisance factor. As in the main text, SimCLR trains a single contrastive loss, similar to Eq[4] but when treating the entire training set as coming from the same single label (ii) Second, No Rec presents our method without using the reconstruction loss (5. Similarly to (Kahana Hoshen 2022). works well in some cases but is not stable across datasets. Especially, in datasets such as Edges2Shoes where invariance very hard to achieve because the domain shifts are dominant.
Table 9: Additional Ablations for the Cars3D dataset (ROC-AUC)
Dataset
Method
AD-Score (1t)
PA-Score (1)
RA-Score (f)
SimCLR
0,780
0,519
0.741
Cars3D
No Rec
0.977
0.511
0.975
Ours
0.985
0.506
0.980
SimCLR
0.805
0.728
0.638
SmallNorb
No Rec
0.779
0.516
0.783
Ours
0.953
0.581
0.943
SimCLR
0.567
0.642
0.510
Edges2Shoes
No Rec
0.55
0.706
0.466
Ours
0.781
0.711
0.719
H TYPICAL STATISTICAL ERROR IN EXPERIMENTAL RESULTS
We ran 3 repetitions of our approach for each experiment. The consistency the results is presented in Tab
I INVARIANCE AND INFORMATIVENESS
We MLP, practical estimator for the nuisance attribute given the code embedding (Tab(TT). As "Optimal" invariance we report the accuracy of predicting the nuisance attribute when information is available all. Although invariance is far from optimal, it is significantly
18

---

Published as a conference paper at ICLR 2023
Table 10: Consistency of the Results Among Repetitions for the Different Datasets (ROC-AUC)
Dataset
AD-Score (f)
PA-Score (()
RA-Score (11)
Cars3D
0,983 + 0.002
0.509
0.011
0.975 + 0.001
SmallNorb
0.952 + 0.008
0.553
0.015
0.947 + 0.008
Edges2Shoes
0.793 + 0.014
0.687 0.019
0.745 + 0.022
Table 11: The invariance and informativeness achieved on the different
Invariance (1) Informativenees(()
SmalINorb
Cars3D
Edges2Shoes
Ours
0.095 0.600
0.123 0.825
0.866 0.716
DN2
0.073 0.023
0.687 0.599
1.000 0.338
MeanShifted
0.068 0.022
0.704 0.618
1.000 0.338
SimCLR
0.575 0.549
0.439 0.745
0.986 0.414
Optimal
0.043 1.000
0.059 1.000
0.500 1.000
better than the invariance achieved by the competing methods. We note that on the SmalINorb dataset, other methods achieve better invariance than our method. However, they are also very low in Informativeness, suggesting that neither nuisance nor relevant attributes are represented with their encoder.
J ALGORITHM BOX
Algorithm 1:
Anomaly Scoring With Labeled Nuisance Attribute
Data: classifier for nuisance attributes C; train data consisting of normal samples Derain' pre-trained encoder f; and a random augmentation function A
Result: An anomaly score S, for each test image
xx Preprocessing:
for v, E Derain do
n, + C(x,)
end
/* Training:
while Not converged: do
Leon = Em,,y, 1 (m) sim (How)))ss)) / I2 E Derain are
arbitrary normal samples
Lree = En,,N &perceptual F2. G(f(x;))
Loug -si (((A((z))A22() A2(zi))
F,G - arg wngming,.(fomm + Free + Laug) /x SGD
end
/* Compute representations for all training samples:
T = (f())x E Derain)
* Anomaly scoring a test sample fy:
fr f(x))
S(x,) ) = ANN Distance (f,,)
K COMPARISON OF DENSITY ESTIMATION METHODS
Following previous anomaly detection works, set the value of to (Cohen & Hoshen 2020).
The results for different values of are shown in the Tab We also compare to alternative density estimation methods (Breunig et al. 2000; Chen al. 2001
19

---

Published as a conference paper at ICLR 2023
Table 12: Comparison of different density estimation methods
Dataset
Method
AD-Score (1))
PA-Score (1)
RA-Score (1))
LOF
0.924
0.576
0.898
SVM
0.597
0.532
0.575
5NN
0,936
0.565
0.928
SmalINorb
1ONN
0,912
0.543
0.909
Ours (INN)
0.953
0.581
0.943
20


Reviews:
[{"review_id": "y7dz2gIUvy", "reviewer": "Reviewer_1pDc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of guiding anomaly detection techniques to ignoring distracting aspects. The distractions or nuisance factors are labeled and the disentanglement is done through contrastive loss. Results on Cars3D, SmallNORDB, and Edges2Shoes are shows that the approach outperforms SimCLR, CSI, DN2, and MeanShifted in terms of AD-Score, PA-Score, and RA-Score. The authors also make the dataset available.", "summary_of_the_review": "The paper addresses an important problem. Problem formulation being influenced by nuisance factors is very relevant.\n\nThe assumptions that users can name nuisance factors and the nuisance factors can be readily labeled are very strong assumptions which are not validated beyond the synthetic data sets explored in the paper. The technical solutions, given the assumptions, are widely known which limit the novelty of the paper.\n\n", "strength_and_weaknesses": "(+) The authors address a core problem and make datasets available.\n\n(+) The technique outperforms unsupervised and self-supervised baselines.\n\n\n(-) Some of the claims in the paper are overreaching: “Taken together in many cases labels for nuisance attributes they can be achieved at virtually no cost.” [Section 4.1] For the approach, the users have to provide additional labeled data. It is not clear if such labels are easily available apart from synthetic datasets.\n\n(-) Anomaly detection applies to various types of datasets. This authors solve it in a specific setting. Does it apply other types of images or to all images or other types of data as well?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T3tBNNsTC6Y", "reviewer": "Reviewer_9jBA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for anomaly detection in the presence of nuisance features.  In their motivation they describe nuisance features as those which give the appearance of anomaly but are not anomalous.  These would be features which are seen in nominal situations as only having a narrow range of values but cannot be realistically expected to be constrained to that range outside of training.  The method requires identification of the nuisance feature and uses a modified loss function with an autoencoder-style network.  The modified loss function is designed to remove the nuisance feature from the reconstruction and to force the reconstructed images to have distributions unaffected by the nuisance feature.", "summary_of_the_review": "A well-written paper featuring an interesting approach.  The approach aims to solve a significant issue with anomaly detection.", "strength_and_weaknesses": "The proposed solution to the identified problem is elegant and creative.  Nuisance features are certainly a problem in real-world anomaly detection applications.\n\nThe example of a nuisance feature in the motivation was not applicable to the approach in that they describe a very high level feature but the example\n\nThe authors are explicit about the strengths and weaknesses of their approach and discuss how the identified limitations can be mitigated.\n\nThe results strongly point to the usefulness of this approach.  However, they are missing a next-level analysis of the results.  It would be nice to see with and without Red PANDA analysis of the role the nuisance feature's role in the reconstruction.\n\nA couple weaknesses: \n* no fault of the authors, is that autoencoders, although popular in the literature, have not been found to be worth the training effort in industry\n* if we have the information needed to provide the training set needed for this approach, then we probably have enough information to create a simple heuristic to deal with these edge cases.\n* I would like to see a more nuanced example of a nuisance feature.  A more real-world example would be features which affect the distribution of the data which can mask anomalies when the nuisance feature is in certain value ranges.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ss_mmvOHdGX", "reviewer": "Reviewer_LAMp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel setting for anomaly detection where the visual data also has a set of attributes that describe the environmental parameters. The proposed method focuses on a setting where the user can define one or more nuisance attributes such that the algorithm should ignore the impact of nuisance attributes when identifying anomalies. This is done by first learning a representation of the data that does not consider the nuisance attribute using a contrastive disentanglement algorithm. The second step is to find anomalies by using a nearest neighbor strategy on the representation data. Results are shown on benchmark data sets and the method is shown to outperform other strategies for detecting anomalies which do not consider the nuisance attribute.", "summary_of_the_review": "The paper has an interesting problem formulation for anomaly detection that I have not encountered before. While there is academic novelty in that, the authors do not quite discuss the practical relevance of the problem setting. There method is reasonable, based on a sound strategy proposed in another paper. There are some issues with the descriptions that could be fixed by a careful rewrite.", "strength_and_weaknesses": "Strengths:\n+ The paper formulates a novel anomaly detection problem which might have potential applications when dealing with visual data.\n+ The idea of using the knowledge of nuisance attributes and incorporating it in the representation phase is based on past idea, but using it for anomaly detection is novel.\n+ The experimental analysis is interesting.\n\nWeaknesses:\n- The approach is meant for a particular type of data sets, for example, images, where the analysis is done on the observations (images) while the nuisance attributes are part of the environmental conditions in which the data was recorded. The nuisance attributes are not encoded in the data as independent features,  otherwise one could just remove them from the analysis. However, the title and abstract does not make that clear. I would request authors to clarify that. Or, if they believe that the method is applicable in other settings (straight tabular data), then they should include discussions about why this would be better than simply removing the attribute.\n\n- The authors distinguish their methods from past work as this - \"Our approach differs from previous approaches to this problem. Previous approaches require knowl- edge of an attribute that should be used to identify the anomalies. Out of distribution detection assumes that anomalies belong to a novel class, outside the fixed set of labelled classes. The weakly supervised setting assumes future anomalies will be similar to a few labelled anomalous samples available during training.\" There are no references here so it is hard to know which approaches are they referring to. Also, I do not think traditional anomaly detection methods require knowledge of attributes to identify anomalies. They possibly require normally labeled data (and maybe some labeled anomalies). Could the authors clarify on this?\n\n- The authors have not talked much about the real world scenarios where this problem formulation might be applicable.\n\n- The level of innovation in the method is limited, given that the method builds on past work on contrastive disentanglement. \n\nSome minor comments:\n- What is an operator? It was unclear in the abstract if the authors are referring to a human operator or a mathematical operator.\n- Is there a citation for this statement - \"A major source for such biases is the presence of additional, nuisance factors.\"?\n- I did not understand the example: \"For example, assume all buses in the training data were captured in a certain pose since they were driving in a bus lane. It is likely that a bus driving lawfully in a regular lane will be viewed from a different angle than the buses in the normal data. Thus, it will be regarded as an anomaly by current standards.\" Are you saying that it is possible that a bus driving lawfully in a regular lane might be viewed from a different angle? I am not well-versed with this domain (and I am guessing many readers won't be as well), so I would request the authors to clarify this.\n- I was confused by this example: \"For example, an encoder guided to be invariant to driving lane, would be trained to contrast images of females against other images of females, but not against images of males (and vice versa).\" Why are we suddenly talking about females and males in the driving context?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QG7UxCfQSeJ", "reviewer": "Reviewer_W97W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper concerns anomaly detection. In particular, it considers the fact that the definition of an \"anomaly\" depends on the use case. The goal is to flag \"real anomalies\" (unusual variation in features we care about) while ignoring \"pseudo-anomalies\" (unusual variation in nuisance features). Nuisance features are labeled in each image. The paper names this setting \"negative attribute guided anomaly detection\" (NAGAD). The paper benchmarks a four baseline methods on NAGAD (3 pre-existing datasets). They also provide their own method, inspired by the domain-supervised disentanglement literature. The new approach has a few components:\n* ImageNet-pretraining for the image encoder.\n* A SimCLR-style contrastive loss applied independently on each collection of images with the same value for the nuisance factor.  This is meant to eliminate information about nuisance factors.\n* A reconstruction loss applied to the output of an image generator. The generator takes as input (i) the representation and (ii) the value of the nuisance variable. ", "summary_of_the_review": "Overall I like this paper and I enjoyed reading it, but I have some questions about the details that I would like to see addressed. There also seems to be a notable disconnect between the stated real-world motivation for the paper and the work in the paper (which is focused on a simplified case with toy problems). Regardless, I find the paper to be valuable and interesting and I would ultimately like to see it accepted if these issues can be resolved. ", "strength_and_weaknesses": "# Strengths\n* Anomaly detection is an interesting and important problem.\n* The paper is generally well-written.\n* The idea that it is more reasonable to specify nuisance factors a priori than it is to specify the features where you expect anomalies makes a lot of sense from a practical point of view, so the NAGAD problem setting seems well-motivated. The metrics used for benchmarking also seem sensible. \n* The experiments are reasonably thorough, with a good number of recent methods tested across several datasets. \n\n# Weaknesses\n* When we consider Table 7 alongside Table 2, it's not obvious that the proposed method is a significant improvement. The proposed method is within the margin of error of a competitor for AD-Score, worse than a competitor for PA-Score (this is true for 2/3 datasets), and better than the competitors for RA-Score. Note that we don't have error bars for other datasets, only for the How should we understand these results?  \n* There is no ablation study for the components of the proposed method. \n* These seem to be toy datasets. If NAGAD is meant to be a well-motivated practical problem, shouldn't there be more challenging real-world datasets where this problem shows up? Section 7 mentions this point, but dismisses it on the grounds that the current methods would not do well. Even if current methods wouldn't work great, it seems worthwhile to ground the study of NAGAD in realistic data. \nOn the topic of practicality, there are a few points relevant to real-world uses that are not addressed:\n* The paper asserts that the (more realistic) case of multiple attributes can be handled \"very naturally\" - I would like a more detailed explanation. \n* The paper assumes that nuisance factors are categorical. Is this a significant limitation or is there an easy way around it? \n* The paper assumes that nuisance factors are always identified correctly - what if the user is wrong about a feature being a nuisance factor? \n* If the user does not enumerate every possible nuisance factor, do they pay a price because the representation is forced to preserve everything else (including all unspecified nuisance factors)?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster