Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective
Authors: João B. S. Carvalho
Mengtao Zhang
Robin Geyer
Carlos Cotrini
Joachim M. Buhmann
Venue: NeurIPS 2023 poster (2023)
Keywords: anomaly detection
causal inference
distribution shifts
TL;DR: Partial Conditional Invariant Representation (PCIR): a novel method grounded in causal inference that leads to AD models robust against multiple distribution shifts.

Abstract: Anomaly detection (AD) is the machine learning task of identifying highly discrepant abnormal samples by solely relying on the consistency of the normal training samples. Under the constraints of a distribution shift, the assumption that training samples and test samples are drawn from the same distribution breaks down. In this work, by leveraging tools from causal inference we attempt to increase the resilience of anomaly detection models to different kinds of distribution shifts. We begin by elucidating a simple yet necessary statistical property that ensures invariant representations, which is critical for robust AD under both domain and covariate shifts. From this property, we derive a regularization term which, when minimized, leads to partial distribution invariance across environments. 
Through extensive experimental evaluation on both synthetic and real-world tasks, covering a range of six different AD methods, we demonstrated significant improvements in out-of-distribution performance. Under both covariate and domain shift, models regularized with our proposed term showed marked increased robustness. Code is available at: https://github.com/JoaoCarv/invariant-anomaly-detection

Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective
João B. S. Carvalho, Mengtao Zhang, Robin Geyer, Carlos Cotrini, Joachim M. Buhmann
Institute for Machine Learning
Department of Computer Science
ETH Zürich (joao.carvalie carvalho, mengtao.. Sanggao.zhang, robin.geyer. ccarlos, jbuhmann)@inf .wt.z. ch
Abstract
Anomaly detection (AD) is the machine learning task of identifying highly dis- crepant abnormal samples by solely relying on the consistency of the normal training samples. Under the constraints of distribution shift, the assumption that training samples and test samples are drawn from the same distribution breaks down. In this work, by leveraging tools from causal inference we attempt increase the resilience of anomaly detection models to different kinds of distribution shifts. We begin by elucidating simple yet necessary statistical property that ensures invari- ant representations, which is critical for robust AD under both domain and covariate shifts. From this property, we derive regularization term which, when minimized, leads to partial distribution invariance across environments. Through extensive experimental evaluation on both synthetic and real-world tasks, covering a range of six different AD methods, we demonstrated significant improvements in out-of- distribution performance. Under both covariate and domain shift, models regular- ized with our proposed term showed marked increased robustness. Code is available at: hattpe//gihabb..oo/DDaaaaaa
com www./oooaarr//iiaaaaaaaaaaaaaaaa wiwwrriiiiiiiiii inttriant ant anomaly detectio
Introduction
Anomaly detectors are the subject of increased interest fields such as finance (Ahmed et al. (2016]. Hilal et al. [2022)), medicine (Schlegl et al. (2019)), and security (Mothukuri al. [2021] Siddiqui ct al. 2019], Hosseinzadeh et al. 2021 ). Having been trained on a sample from an unknown
distribution, these models capable of identifying abnormal objects unlikely to come from the original distribution (Bishop and Nasrabadi] (2006)). Anomaly detection (AD) stands apart from supervised classification
does not involve anomalies during training, making it challenging articulate a model that depicts the class of objects deemed as normal.
AD as a field boasts plethora of diverse methodologies (Ruff et al. 20211). Current detectors have demonstrated the advantage of approaches based on representation learning (Reiss and Hoshen| [202T],
Deng and Li 2022). ). In this context, encoder maps objects to representations which capture the most distinctive features of an object. In addition, it strives to map the class normal objects onto subset characterized by more regular shape, thereby rendering representations from abnormal samples easily identifiable by comparison. Central to this second goal a notable vulnerability of representation learning-ban based methods: they hinge the assumption of independent and identically distributed (i.i.d.) training and test data. This implies that normal samples the training data are ex- pected be sampled identically in the test data, thereby being mapped to the same vicinity in the repre- sentation space an assumption that is frequently violated in real-world scenarios (Koh et al. (202
Indeed, distribution shifts in the context of AD present unique challenge because it involves discem- ing two types of distribution shifts targeting the distribution of the normal objects, Pa- Anomalies
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

can be conceptualized as samples from different distribution Pa. which scarcely overlaps with the mass of Pt.- This distribution Pa, can be interpreted as shifted variant of Pu. and the task of anomaly detection becomes identifying this shift. Changes in the environment can also produce distribution shifts. When such a distribution shift transpires from the training data to the test data, Ph is transformed into Pn. If anomalies are naively perceived as merely samples from the original normal samples that are sufficiently different, then all samples from P., could be wrongly classified as abnormal samples. Therefore, it is crucial to note that anomaly detectors need to identify shifted versions of Pn that generate anomalies while being resilient to those originating from any other Pn.
Additionally, anomaly detectors are particularly susceptible to changes in features that are incidentally correlated with being normal. This is because the training data consists only of normal objects, which in conjunction with consistent exposure to any recurrent features present in the environment, inherently facilitates the emergence of shortcut learning (Geirhos et al. (0200) ). These effects can significantly impact the reliability of crucial anomaly detectors. For example, consider an anomaly detector designed for monitoring bank transactions, predominantly occurring during business hours. Such a detector could mistakenly hinge on the transaction time to determine anomalies. This approach could lead to fraudulent transactions executed at noon going undetected, while regular transactions made by individuals active at night might erroneously be flagged as suspicious. Thus, the effective detection of anomalies requires the model to go beyond such superficial correlations and instead focus more substantive, causally relevant features.
In this work, we formalize the problem of anomaly detection in the presence of domain and covariate
shifts using causality and information theory ( (Wang and Veitch [2022] Wang et al. [2022), .. Jiang
and Veitch [022]. Liu et al. [202T], [insker 1988]. SStnn 2004]. Hjelm al. 2018). We view the causal graph and distinguish between the environment that generates the dataset, the content features, the style features, and representations that are computed from these features. We note that to mitigate vulnerabilities to distribution shifts, we need to produce representations that are invariant to the environment where the object comes from (Veitch et al. [021]] Wang et al. [2022), Wang and Veitch (022)). ) That is, the representation is not causally influenced by the environment. Through this formalization, we derive and formally justify a necessary condition for learning invariant representations within anomaly detection. To impose this condition,
generation of the dataset
introduce
the partial conditional invariant regularization (PCIR) term for learning invariant representations in this setting. This term minimizes discrepancies between representations from different environments by minimizing discrepancies like the maximum mean discrepancy (MMD) (Gretton et al. (2012)). Similiar approaches relying MMD regularization for domain generalization in different tasks have proven to be valuable (Kang et al. 22019. Long et al. (2013)). However, to our knowledge, all methods relying on conditional invariance have been applied under the assumption of access to all classes during training. This is not the case in the AD setting.
Contributions: (1) We conduct series of experiments to illustrate the limitations of current AD methods in handling various distribution shifts. (2) Through causal inference we propose novel formalization of the dual requirements of informativeness and invariance for robust AD models, leading to new insight, as described in Theorem 2 (3) We introduce regularization term to induce partial conditional distribution invariance, which serves as a strategic measure to ensure robust AD amid distribution shifts. (4) Our empirical findings demonstrate the effectiveness of our regularization term; all tested models exhibited an enhancement in performance under both covariate and domain shifts when augmented with purtial conditional invariant regularization (PCIR).
2
Related Works
2.1 Distribution Invariant Representation Learning
The relationship between invariance and robustness to shifts in data distribution has been extensively explored, notably within the field of causal inference.
range of studies has pursued the development of invariant representation learning, achieving notable
success (Long et a [0033] Kang et al. [2019) Mitrovic et al. 20200] Lv et al. [2022]. Nguyen et al.
(02TT). Moreover, detailed examination of various forms invarianc--baee methods derived
from underlying causal graphs has been provided by Veitch et al. 2021] and Wang and Veitch 2022

---

Pz
Boee))
Px
Boc)
Elo(Eme))
po (meaa)
b
invariant and in formativee
23
(ime))
po B=
or original mulb--neerrommnnt conj conformate formation
Ps
Pz
pytat Eimea)
bustriaatnnn ovr the normal class
P
auuples
Paimarrual sucpples'
yout (lotEmms)
(c)
inveriant but non-in informatiee
(d)
in. oormattiv but Ron-inooriann
Figure 1: Comparative analysis of various mapping strategies from X' to E in the context of AD under distribution shifts. (a) Anomaly detection setting with multiple environments in X. (b) Ideal scenario where the mapping is both invariant and informative; normal samples from different environments converge to the same subset of Z, maintaining the distinction between normal and abnormal samples. (c) Mapping is invariant but not informative, resulting in both normal and abnormal representations collapsing to the same subset of Z. (d) Mapping is informative but not invariant; although the mappings of all environments remain distinct, this makes the encoder vulnerable distribution shifts.
Causal inference has also been shown foster distributional robustness, as documented in Mein shausen [2018). Building on this notion, Invariant Risk Minimization (IRM) (Arjovsky et al. [2019]
introduced a novel optimization principle hinged on maintaining invariance across diverse environ- ments. However, Koh et al. [2021] demonstrated that this approach tends to falter when subjected to
real-world distribution shifts. Yao al. [2022] has introduced selective data augmentations to induce
distribution shift robustness. However, due to its reliance on class labels, it is not directly applicable AD.
2.2 Deep Anomaly Detection
A common strategy in AD involves the deployment of autoencoders. These methods operate by learning normal patterns within a dataset and then attempting to reconstruct new samples. The premise is that anomalous samples, due to their distribution differences, will yield higher reconstruction errors. Several studies have adopted this method, demonstrating its efficacy in a range of contexts (Gong
et al. 2019 Park et al. 2020 Yan et al. (021))
In the wake of recent advancements in self-supervised representation learning (He al. (019
Chen et al. (2020)), contrastive learning has emerged as a viable method for AD. Approaches such as PANDA [Reiss et al. (2021)), and the more recent MeanShift (Reiss and Hoshen (2021]) and CFA (Lee et al. 22022) harness these strategies to finetune pre-trained encoders to the AD setting. Conversely, CSI (Tack et al. (020) ) and DROC (Goyal et (2020]) use contrastive learning to learn representation of the normal data from scratch. Knowledge distillation-base methods also hold a significant place in AD research. student-teacher framework, which incorporates discriminative latent embeddings, was first introduced in Berrgmann et al. [202)) The most recent iterations, Reverse Distillation (Deng and Li 2022) ) and STFPM (Wang et al. [202T] have attained state-of-the-ar performance across numerous AD tasks. Recently, Red Panda (Cohen et al. (2023)), has adapted PANDA by adding an additional disentanglement term to the loss derived from Kahana and Hoshen 2022 with the intent of inducing robustness to previously identified attributes.
3

---

Despite the effectiveness of these methods, it is important to note that they primarily focus on AD within datasets where the training and test data distributions are identical. As we will show, its performance tends to degrade when confronted with data that exhibit different kinds of distribution shifts.
One way to improve robustness to distribution shifts entails pretraining the encoder using an invariance-inducing method (for example, IRM or LISA), applied using the same data (Smeu et al. 2.22 Unfortunately, this requires the existence of additional task labels, which is uncommon for AD tasks. Moreover, as we experimentally demonstrate, this approach achieves limited performance when paired with state-of-the ahe-trt-at AD methods.
unrelated pretext task
Formalization
3.1 Background
Our formulation operates under the assumption of a given set of environments, denoted &, and a sample space, X. We have a sample D {.1,.... Fn. drawn from an unknown distribution Pn. whose mass mostly concentrated on subset N X. This subset, N, defines the normal class. Given that X ~ Pn. we consider X as pair of random variables Xa and Xe. such that X = (Xa, X.). Here, the environment E only influences X.. Conceptually, Xa represents the component of X that determines whether X is an anomaly, while X, comprises style features. These style features, while unaffected by the anomaly status of object, are influenced by the environment. Note that this assumption implies that X' = N x Xe, for some appropriate sets X. and Ne. Fig. (a) illustrates our setting under three different environments.
Our main goal in this section is to elicit the requirements that representations should fulfil in order to lead effective anomaly detectors. More precisely, argue that representations must simultaneously (i) maximize the information they contain about the original objects and (ii) they must be invariant to the environment.
3.2 Requirements for Robust Encoders
We illustrate the necessity of invariance and informativeness for effective AD using an example with random variables X1, X2, X3. We assume that they are distributed over the set N of all points (x1, 22, F3)) E RB3 such that 0 S 21 = x2 S 1. We assume that Ty denotes the environment. Our dataset D consists of points in N where 23 = 0.
Consider now the following encoders fi. f2. and f3. where fi x2, = T1. fi ( 22 23) (x1, 22), and (21, F3) = (31, 22, Tz). Note that encoder fi is invariant but lacks the necessary information to detect anomalies, as can map different points to the same representation. Indeed, the normal point (1, 1. 1) and the anomaly point 21, 1) would be mapped to the same representation, the number 1. Hence, any anomaly detector based on this encoder can be easily evaded. Encoder fa. captures all information but is not invariant, which leads to potential evasion of AD, Encoder /2, on the other hand, is both invariant and informative, correctly capturing the necessary information to detect anomalies while ignoring the environment variable .33
This simplified scenario emphasizes the dual necessity of invariance and informativeness for the development of robust anomaly detectors. Regrettably, as we will demonstrate, contemporary AD methods tend to excel in informativeness while significantly lagging in invariance. For graphical illustration of this effect, please see Fig. (b-d).
We now formalize the desiderata for representations computed by an encoder for AD.
3.3 Invariant Representations
Let f be an encoder mapping a X' a representation space Z. We also let Z f(X) denote the representation of X.
Definition 1. We say that Z is an invariant representation of X under domain shifts if
do (Z :) =000E==0) =)., for any e E E.
(I)

---

Definition 2. We say that Z is an invariant representation of X under covariate shifts if
f - =)), for any x, E X..
We simply say that Z is an invariant representation of X it remains unaltered under both domain shifts and covariate shifts. Note that domain shifts are stronger than covariate shifts. Specifically, domain shifts entail assessing data from diverse sources with disparate parameters, whereas covariate shifts pertain to subtle alterations in the existing data. Consequently, the broader changes inherent domain shifts usually exert more substantial impact on data representation stability and are harder tackle.
Invariant representations prevent anomaly detectors from incorrectly classifying objects that result from covariate and domain shifts. These representations are, by definition, resistant changes in E or Xe. As a result, adversaries cannot expect to alter the detection an anomaly by merely using a different environment changing style features of the anomaly.
However, solely enforcing invariant representations can be insufficient, and potentially lead to collapse of the representations (see Fig. (c)) For this reason, we argue that representations shall store as much information about the original object as possible, This principle is line with the
design of many popular encoders (Linsker 1988 Stone 2004 formalized using information theory.
Hjelm et al. (2018) and can be
3.4 Mutual Information between Representations and Objects
We can use the mutual information between two random variables as our metric to evaluate how much information about X is captured by the representation Z.
Definition 3. The mutual information (Xx; 2) between X and Z is defined
I(X; Z) ʃ
px (x, 2) Px (x (z)
dx dz.
Here, Px,2 is the joint pdf of (X, 2) and Px and pz the marginal pdfs of X and Z, respectively.
Intuitively, (X; ; Z) indicates how much information learning about one X or Z reveals about the other. We argue then that encoders shall compute representations that keep much information as possible from the original objects.
Learning Invariant and Informative Representations
Having outlined the prerequisites for anomaly detection representations, the next task is to devise strategies that enable encoders to learn both invariant and informative representations. Encouragingly. most contemporary AD methods inherently employ strategies that maximize the mutual information between the observation and the representation (Tishby and Zaslavsky (2015).. This is often achieved either via reconstruction loss term (Kong et al. [2019]) or contrastive loss term (Sordoni al. [
Wu et al.) 22200) Oord et al. 2018
We dedicate the rest of this section developing strategy suitable for invariant representation learning under the scope AD.
4.1 Learning Invariant Representations
We start by introducing causal graph for anomaly detection in Fig. In these graphs, E, X.. and X, denote the environment, the relevant features, and the style features. We introduce two new random variables W and U. W is binary variable indicating if the object is normal (W = 0) or an anomaly (W = 1). U denotes all confounding factors. That is, factors that produce correlations between X. and X. during the sample generation process.
now recall that an encoder I that attains invariant representations X,-measurable (Viichh et al. 2 The following result, which follows from theorem from Veitch et al. [2021] argues that we can ensure invariant representations by enforcing possibly conditional statistical independence between Z and E.
5

---

U
U
W
E K
W
E
K
X
X T
X
X T
Z
Figure 2: Causal graphs for anomaly detection. The left figure shows the case of no confounding. The right figure shows the case of confounding. intervention the E variable induces a domain shift (gray hammer), whereas intervention at the Xe variable induces a covariate shift (black hammer).
Theorem Suppose that f learns invariant representations. W and E are confounded, then ZLL E W. Otherwise, Z E.
Proof. We sketch a proof here and provide more rigorous proof in the appendix. Since / learns invariant representations and is, therefore, measurable, we can say that Z is X. -measurarbe. This means that any probability of an event involving can be rewritten as probability of an event involving X.. As result, it suffices to show that X - E W, when W and E are confounded and X, .LL. E, otherwise. These claims can be shown using d-separation. For example, when W and E are not confounded, there are only two paths from X, to E and they are blocked. Hence, X E by d-separation.
The fundamental question we must address how to ensure statistical independence between Z and E. such independence holds, we find that for e, e E &, it implies that pdo(E=e)( (Z =) pdo(E=e)) (Z = ). In general, we cannot access counterfactual examples, so enforcing counterfactual invariance becomes impossible. However,
still possible induce counterfactual invariance signature by imposing appropriate conditional independence conditions. In practice, this condition -
E e')) O, where MMD stands for maximum mean discrepancy (Gretton et al. 2006 2012)). ) and measures the distance between two distributions using empirical samples by calculating the divergence between the means of these sample sets once they have been projected into reproducing kernel Hilbert space (RKHS).
can be MMD (p(2=11 E e), p(Z
4.2 Partial Conditional Invariant Regularization (PCIR)
Based on the previous insights, we propose using the MMD between two distributions as the driver for the wwarrance-inducing regularization term. Taking into account previous formulation depicted Fig. 2 we are now in a position to derive a novel regularization term specifically designed for invariant AD:
OPCIR =
Σ MMD (p(Z = W = 0, = = 0,
(4)
e.e'E8
egfe"
We call this approach partial conditional invariant regularization (PCIR), as it induces conditional distribution invariance over only instantiation of W.
This partial conditional regularization aligns with other regularization terms proposed preceding research (Veitch et al. 2021 Li et al. 2018). ). It is "conditional" because, in the event of confounding. we must condition on W when computing the MMD. It is 'partial' due to the fact that the training dataset only contains samples where W 0.

---

5 Experiments
5.1 Experimental Setup
Our solutions are validated under two distinct settings: domain generalization across domain shifts and domain generalization across covariate shifts. All experiments described were the result of repetitions over different seeds. For additional details on the experimental design please refer to the supplementary material.
Real-world domain shift For a realistic anomaly detection scenario, we considered the task of identifying tumorous tissue from images of histological cuts, using the Camelyon17 (Koh al.) [202T] Bandi et al. (2018) dataset. This dataset consists of five different subsets of images arising from five different hospitals, with domain shifts occurring due to differences in slide staining, image acquisition protocol, and patient cohorts. This presents challenging anomaly detection task, as the alterations that differentiate normal from abnormal samples are often subtle and may correlate with unknown features that are domain-dependent. In our experimental design, we motivate an environment per domain and set up two sets of experiments: one using three environments for training and another using two environments for training.
Real-world shorteut We have also evaluated our method in the Waterbird dataset (Sagawa et al. (2019). This is real-world natural image dataset where the distribution shift occurs the natural habitat depicted in the background changes between an aquatic and land setting. From the two kinds of birds in the dataset (water and land birds), water birds were assigned to training data and land birds were set as an anomaly. To make this a more challenging setup, we have defined a highly unbalanced distribution of the environments, with 184 images in training data with water background and 3498 with a land background. The evaluation of the methods was performed in images with only a water background for cat-of-diisributtio and only land background for in in-distribution.
Synthetic covariate shifts dataset Eulig et al. (202T] for our experiments. This dataset comprises modified and upsampled images from MNIST Deng [012], where the generative factors of the image can be altered, leading to changes in texture, hue, lightness, position, and scale.
To evaluate the robustness against covariate shifts, we use the DiaViB-6
The training data is comprised of two distinct environments, generated by manipulating original instantiations of each factor, and ensuring all factors differed between the two environments. For the test data, we defined five distinct environments denoted as eg, El, E4. The environment eo corresponds to images from MNIST for specific digit under an original instantiation of each factor. All images of that digit are labeled as normal, while images of any other digit are considered anomalies. For 0 < i < 4, each environment e, corresponds to images where i factors have been modified. In e; all factors modified in 6;-1, plus an additional unique factor, are altered. The task is to classify an image of a handwritten digit as either normal an anomaly.
We also adapted this setup to include the MaSSI dataset, a more challenging synthetic benchmark. The same cumulative covariate shifts were induced from an original in-distribution environment, en, to subsequent environments 61-4- Two specific classes were chosen to generate normal and abnormal samples.
Anomaly detection methods tested We undertook comprehensive evaluation of our proposed regularization term, PCIR, and integrated it into six diverse methodologies for deep AD. These include: STFM Wang et al. [202T]. Reverse Distillation Deng and Li [202]] CFA [ee et al. 2022] MeanShified Reiss and Hoshen [022]) CS/ Tack al. [202)] and Red PANDA Cohen et al. 2023 For a detailed exposition of these methods, refer to the supplementary material.
5.2 Results
Real-world anomaly detection under domain shift Notably, even with the complexity and real- world variability introduced by the domain shift, the incorporation of partial conditional distribution invariance still resulted in notable improvements for both training setups, in particular with MeanShifi and Red PANDA retrieving almost in-distribution performance (see Fig. 3) This suggests that our regularization term is robust to more substantial changes associated with domain shifts.
7

---

PCR
10
000
(a) Camelyon 17: 2 environments
(b) Camelyon17. 17: environments
(c) Waterbirds
Figure 3: Results on realistic domain shift Camelyon17 and realist shorteut learning in Waterbirds. (background transparent bar-plots: in-distribution evaluation; foreground opaque bar-plots: out-of-distribution evaluation). (a) Camelyon17: Setup with two in-distribution training environments and three out-of-distribttion test environments. (b) Camelyon17: Setup with three in-distribution training environments and two out-of-distribution test environments (c) Waterbirds: Shortcut learning setup.
Focusing the out-of-distribution setting, our results highlight a consistent pattern of performance enhancement when applying the partial conditional invariance regularization term, ranging from 2% to 8% increase in the AUROC when comparing regularized and un-regularized methods. The consistency of this performance boost across different models and with both two and three in- distribution training environemnts underscores the potential of PCIR beneficial regularization technique in out-of-distribution settings.
Realist shorteut learning We observe that in all original unregularized methods evaluated, there is a noticeable drop in performance when comparing in-distribution (background transparent plots) to of-distribution (foreground opaque plots) that attributed to the models effectively capturing the exposed shortcut. Models with PCIR nearly recover the in-distribution performance, showing the effectiveness of this approach to ignore uninformative environment features even when exposed through a shorteut. Furthermore, in all models tested, there's consistent observation that the model performance not only stabilizes but also increases when adding the regularization term even for in-distribution scenarios. The increase in model performance, attributed to the addition of PCIR to each method, exhibits an improvement range from 5% to 15% AUROC
MMNIST and Fashion- under covariate shift Models that absorb shorteut features have been observed to be especially susceptible to covariate shifts (Eulig et al. [2021]. Geirhos et al. ) ). Therefore, if a model abuses a shortcut, then inducing a covariate shift in the shorteut feature often significantly deteriorates performance. This effect is discermible in Fig. 4 where performance drops progressively increase for all models more covariate shifts are induced in the images, and thus more shortcut features deviate from their original form in the training data.
However, note that by adopting regularization based partial conditional invariance, we have been able to construct anomaly detectors that consistently exhibit enhanced robustness against induced covariate shifts. In some cases, even in-distribution performance increases through partial conditional invariance. These findings corroborate our hypothesis that partial conditional distribution invariance serves a sufficient prior for robustness covariate shifts in AD. For a more comprehensive analysis of performance improvements achieved refer to the supplementary material.
Ablation study In our ablation study, we held all method parameters constant to exclusively examine the impact of the weight assigned to the partial conditional invariance regularization term.
Across all tested methods, the relationship between regularization weight and performance exhibited a concave shape. This pattern suggests a simple linear search could be sufficient to identify an optimal weight for the regularization term. This observed behaviour aligns intuitively with the nature of invariance regularization. Over-emphasiziee model's ' encoder to generate non-diseriminative and non-informative representations. In an extreme case, the model could collapse all inputs into a single value. Interestingly, the behaviour of the regularization term remained consistent across both datasets used in our study. This reinforces the robustness of our approach across diverse datasets.
the regularization term could potentially cause the
8

---

008
000
9399 Lomeetardd
PANIM
53151
PANIM
oup
83999 lnmeelaerd
PANIM
(a) c1: MNIST s.t. single covariate
o2: MNIST s.t. double covariate
(c) 63: MNIST s.t triple covariate shift
shift
shift
PCR
3oc
oa
022
(d) e4: MNIST s.t. 4-fold covariate
(c) e1: MaSSI s.t. single (f) 62: Fashion-MNIST s.t. double
shift
covariate shift
covariate shift
179
(g) Fashion-MNIST triple (h) e4: FasHion-MNIST S.I. 4-fold covariate shift
covariate shift
Figure 4: Experimental results on synthetic covariate shift (background transparent bar-plots: in- distribution evaluation; foreground opaque bar-plots: out-o--distribution evaluation) (a-d) Results in MNIST. (e-h) Results in Fashion-MNIST
0,70
InaanAUROC
0.00
0..00
0.00
0.50
Mandark
0.50
CB
CSI
0.0
0.001
0.01
0.1
1.0
10.0
100.0
0.0
0.001
0.01
0.1
2.0
10.0
100.0
Regularisation wwight
Regularication gla
(a) MNIST
(b) Fashion-MNIST
Figure 5: Ablation study over the weight of the regularization term for MNIST and MaSSIO under distribution shift. Regularization weight determines the weight of PCIR in the training loss.
6 Discussion and Limitation
Invariance regularization also improves in in-distribution AD A surprising result from this study was the consistent enhancement in-distribution performance across most methods when partial conditional invariance was incorporated. This observation can be explained by recent advances suggesting that invariance also bolsters in-distribution robustness against noise and sampling variabil- ity (Lopes et al. 2019)) Mitrovic et al. (2020)) factors to which anomaly detection is inherently vulnerable (Ramotsoela et al. 2018)). ). By design, invariance regularizers discourage representations encoding style features. Hence, representations only carry over information that is inherently related to the object rather than the environment. This leads models identifying meaningful patterns instead of noisy variations in the data, even in in-distribution.

---

Unlabeled environments A prominent limitation of invariance regularization lies in its dependence on environment labels (Veitch et al. [202T). Jiang and Veitch| [022], , Arjovsky et al. (2019). Addition- ally. as evidenced in our experimental comparisons between covariate and domain shifts, interventions directly applied to the covariates during environment generation yield enhanced robustness under the constraints of partial conditional invariance. This underscores that conditional invariance regular- ization's 's effectiveness diminishes when the environments are not distinctly segregated. However, in situations where datasets lack explicit environment partitions, alternative strategies can be employed. As demonstrated by Lin et al. [2022] remains feasible to jointly estimate environment partitions and invariant representations, provided there is access to sufficient auxiliary information. This finding opens the door to expanding the applicability of partial conditional invariance in AD, even in scenarios with limited information on environmental conditions corrupted separation between environments.
Invariance beyond partial conditioning Theorem shows that in the presence of confounding, learning invariant representations requires that representations independent from the environment, when conditioned on W. Note that our partial conditional invariant regularization conditions only on W = 0 and not W 1 (see Equation 4. This is an inherent limitation due to the fact that only have normal objects in the sample. However, our regularization is still powerful enough to provide improvements over the state of the art. We argue that the quality of this improvement depends on how disentangled content features are from style features (i.e., Xa and X, in Figure 2) Consider, for example, an MNIST setting where normal objects are images of particular digit and images of any other digit are anomalies. Furthermore, assume that images from one environment have one background color and images from another environment have another background color. There is clear disentangiement between style (i.e., the digit) and content (i.e., the background color). In such settings, attaining invariance to the background color just with W leads also to invariance to the background color with W = 1. We conjecture that partial conditional invariant regularization is sufficient when W does not influence Xg: that is, when there no arrow between these variables in Figure 2
Multi-shift environments Certain datasets, such as the one described by Christie et al. (0018) take into account data shifts across two different kinds of domain shifts (e.g. time and location), equivalent to dual interventions bivariate E in our formulation. While the extension of our work to address multi-attribute settings might be plausible in a dual intervention scenario. it presents non-trivial challenge as the number of different possible interventions increases, given that requires the induction of pairwise invariance across environments. This issue continues to be an area of active research.
Fairness One potential extension of this work concerns the setting where either covariate shift induced over a protected attribute (e.g., gender or race), or a domain shift that permeates to such attributes. Such nuances have been brought in the context of invariant representation learning by Veitch et al. [2021] and could serve as a potential application of this work in AD. In particular, we regularization term be implemented similarly to Louizos et al. [2015), but instead with the intent of effectively discarding incidental correlations that might reflect societal or systemic biases present in the data.
could
7 Conclusions
Our study sheds light on the significant challenges anomaly detectors face in the context of distribution shifts. We proposed novel, robust solution centered around invariant representations, which mitigates the impact of shortcut learning by enforcing statistical independence between the representations and the environment. Empirical validation of our theoretical proposals confirmed the effectiveness of our approach, with a regularization term inducing partial conditional distribution invariance, significantly improving model performance under covariate and domain shifts. We believe these findings pave the way for deeper understanding of AD methods" robustness and how to mitigate their vulnerability to distribution shifts.
10

---

References
M. Ahmed, A. N. Mahmood, and M. R. Islam. survey of anomaly detection techniques in financial domain. Future Generation Computer Systems, 55:278-288, 2016.
S. Akcay, D. Ameln, A. Vaidya, B. Lakshmanan, N. Ahuja, and U. Genc. Anomalib: A deep learning library for anomaly detection, 2022.
M. Arjovsky, S. Chintala, and I.. Bottou. Wasserstein generative adversarial networks. D. Precup and Y. W. Teh, editors, Proceedings the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 223. PMLR, 06-11 Aug 2017. URL https: httpps://procceeinggg.. mlr. press/v0//arppep /ttes///////arjoosyyaa......
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 190.002997 2019.
P. Bandi, O. Geessink, Q- Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, B. E. Bejnordi, B. Lee, K. Paeng, A. Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 38 (2):550-560, 2018
P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4183 4192, 2020.
C. M. Bishop and M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Chen, Kornblith, M. Norouzi, and Hinton. simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
G. Christie, N. Fendley, J. Wilson, and R. Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172-6180, 2018.
N. Cohen, J. Kahana, and Y. Hoshen. Red PANDA:
Disambiguating image anomaly detection by removing nuisance factors. In The Eleventh International Conference on Learning Representations, 2023.
H. Deng and X. Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9737-9746, June 2022.
I. Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2966):141-142, 2012.
E. Eulig, P. Saranritichai, C. K. Mummadi, K. Rambach, W. Beluch, X. Shi, and V. Fischer. Diagvib- 6: diagnostic benchmark suite for vision models in the presence of shorteut and generalization opportunities. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10655-10664, 2021.
R. Geirhos, J. Jacobsen, C. Michaclis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):65--673 2020.
D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh, and A. v. d. Hengel. Memorizing normality detect anomaly: nemory-augmenter deep autoencoder for unsupervised anomaly detection, In IEEE International Conference on Computer Vision (ICCV),
S. Goyal. A. Raghunathan, M. Jain, H. V. Simhadri, and P. Jain. DROCC: Deep robust one-class classification. In International conference on machine learning, pages 3711-3721. PMLR, 2020.
A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the two- sample-problem. In Schölkopf, J. Platt, and Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.

---

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 2019.
W. Hilal, S. A. Gadsden, and J. Yawney. Financial fraud: A review of anomaly detection techniques and recent advances. 2022.
R. D. Hjelm, A. Fedorov, Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808. 06670, 2018.
K. Grewal, P. Bachman, A. Trischler, and Y. Bengio.
M. Hosseinzadeh, M. Rahmani, Vo, M. Bidaki, M. Masdari, and M. Zangakani. Improving security using SVM- anomaly detection: issues and challenges. Soft Computing, 25:3195- 3223, 2021.
Y. Jiang and V. Veitch. Invariant and transportable representations for anti-causal domain shifts. arXiv preprint arXiv:2207.01603 2022.
J. Kahana and Y. Hoshen. A contrastive objective for learning disentangled representations. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 579-595. Springer, 2022.
G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark in-the-wild distribution shifts. International Conference on Machine Leurning, pages 5637-5664. PMLR, 2021.
L. Kong, C. d. M. d' d'ttumee W. Ling, L. Yu, Z. Dai, and D. Yogatama. A mutual information maximization perspective of language representation learning. arXiv preprint 2019.
S. Lee, S. Lee, and B. C. Song. CFA: Coupled-hyperspherr--aess feature adaptation for target- oriented anomaly localization. arXiv preprint arXiv:2206.04322 2022.
H. Li, J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018
Y. Lin, S. Zhu, L. Tan, and Cui. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35:24529-24542, 2022.
R. Linsker. Self-organization in a perceptual network. Computer, 21(3):105-117 1988.
C. Liu, X. Sun, J. Wang, H. Tang, T. Li, T. Qin, W. Chen, and T.-Y. Liu. Learning causal semantic representation for out-o--distribution prediction. Advances Neural Information Processing Systems, 34:6155-6170 2021.
M. Long, J. Wang. G. Ding, J. Sun, and P. S. Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013.
R. G. Lopes, D. Yin, B. Poole, J. Gilmer, and E. D. Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv: 026/1, 2019.
C. Louizos, K. Swersky, Li, M. Welling, and R. Zemel. The variational fair autoencoder. arXiv preprint arXiv:1511.00883 2015.
F. Lx, J. Liang, S. Li, B. Zang, C. H. Liu, Z. Wang, and D. Liu. Causality inspired representation learning for domain generalization. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8046-8056, June 2022.
12

---

N. Meinshausen. Causality from a distributional robustness point of view. In 2018 JEEE Data Science Workshop (DSW), pages 6-10. IEEE. 2018.
Y. Ming, H. Yin, and Y. Li. On the impact of spurious correlation for out-ofddistribttion detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10051-10055 2022.
J. Mitrovic, B. McWilliams, Walker, L. Buesing, and C. Blundell. Representation learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922,, 2020.
V. Mothukuri, P. Khare, R. M. Parizi, S. Pouriyeh, A. Dehghantanha, and G. Srivastava. Federated- learning-ban based anomaly detection for IoT security attacks. IEEE Internet of of Journal, 9()): 3545-25544 2021.
A. T. Nguyen, T. Tran, Y. Gal, and A. G. Baydin. Domain invariant representation learning with domain density transformations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5264-5275 Curran Associates, Inc., 2021. URL http::::/ppoceeeeeiiiiiieee/ nwwrips...ppp...... seuripp.cc/ppper. fi
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0000110770304 2018.
H. Park, J. Noh, and B. Ham. Learning memory-guided normality for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14372 14381, 2020.
D. Ramotsocla, A. Abu-Mahfouz, and G. Hancke. A survey of anomaly detection in industrial wireless sensor networks with critical water system infrastructure as a case study, Sensors, 18(8): 2491, 2018.
T. Reiss and Y. Hoshen. Mean-shifted contrastive loss for anomaly detection. arXiv preprint (it:22/06.08844, 2021.
Reiss, N. Cohen, L. Bergman, and Y. Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2806-2814, 14, 2021.
L. Ruff, J. R. Kauffmaan, A. Vandermeulen, G. Montavon, W. Samek. M. Kloft, T. G. Dietterich. and K.-R. Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-795, 2021.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
T. Schlegl, P. Seeböck, S. M. Waldstein, G. Langs, and U. Schmidt-Erfurth. f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis, 54:30-44, 2019.
M. A. Siddiqui, J. W. Stokes, C. Seifert, F. Argyle, R. McCann, J. Neil, and J. Carroll. Detecting cyher attacks using anomaly detection with explanations and expert feedback. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2872-2876. IEEE, 2019.
S. Smeu, E. Burceanu, A. L. Nicolicioiu, and E. Haller. Env-aware anomaly detection: Ignore style changes, stay true to content! arXiv preprint arXiv:2210.03100 2022.
A. Sordoni, N. Dziri, H. Schulz, G. Gordon, P. Bachman, and R. T. Des Combes. Decomposed mutual information estimation for contrastive representation learning. In International Conference on Machine Learning, pages 9859-9869. PMLR, 2021.
J. V. Stone. Independent component analysis: a tutorial introduction. MIT press, 2004.
Tack, S. Mo, Jeong, and J. Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
13

---

D. Tellez, G. Litjens, P. Bándi, W. Bulten, J.-M. Bokhorst, F. Ciompi, and J. Van Der Laak, Quantify- ing the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, 58: 101544, 2019.
N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle, In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
V. Veitch, A. D' Amoor, S. Yadlowsky, and J. Eisenstein. Counterfactual invariance to spurious correlations: Why and how to pass stress tests. arXiv preprint
2021.
G. Wang. S. Han, Ding, and D. Huang. Student-teacher feature pyramid matching for anomaly detection. In The British Machine Vision Conference (BMVC), 2021.
R. Wang, M. Yi, Z. Chen, and S. Zhu. nut-of-distribution generalization with causal invariant transformations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 375-385, 2022.
Z. Wang and V. Veitch. A unified causal view of domain invariant representation learning. arXiv preprint arXiv:2208. 20000297778 2022.
M. Wu, C. Zhuang, M. Mosse, D. Yamins, and N. Goodman. On mutual information in contrastive learning for visual representations. arXiv preprint 10xxv::0005131444 2020.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.0771 2017.
X. Yan, H. Zhang, X. Xu, X. Hu, and P.-A. Heng. Learning semantic context from normal samples for unsupervised anomaly detection. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3110-311 May 2021. doi: 10. 10.1609/aaai..o w.t....1544...44.... URL https:////// aaai org/
index. phpp hbtp/AMt/artiile//iee// 1e4214200
H. Yao, Y. Wang, S. Li, L. Zhang. W. Liang. J. Zou, and C. Finn. Improving dut-of-distribution robustness via selective augmentation. In Proceeding of the Thirty-ninth International Conference on Machine Learning, 2022.
14

---

A Theoretical Details
A.1 Measurability
Let B (RR) be the Borel o-algebra over R", for some n E N. Definition 4. The d-algebra generated by random variable Y is o (B) B E B (R))). Definition 5. A random variable X is Y -measuraalle if o C (Y))
Intuitively, the o-algebra generated by Y describes all events that Y "can express" and can be measured probability. If X Y measurable, that means that Y can express all what can express.
A.2 D-separation and Confounding
We provide here brief overview on d-separation and confounding and refer the reader to Bishop and Nasrabadi 2006) for details.
Definition 6. Two random variables in Bayesian network confounded if they share latent parent.
Definition 7. A path a sequence of random variables (Vi...., V2) a Bayesian network, where V, is parent child of V4+1. for i < n. For 1 < i < the variable V; is a collision if V, is child of both Vi-1 and Vi+1- Definition 8. A path blocked if V1 and V, are not observed and least one of following holds:
For some collision V in the path, neither V nor any of its descendants is observed.
For some variable V in the path that is not a collision, V is observed.
We now recall the d-separation principle.
Lemma Let W be a set observed variables and V1 and V/2 two variables not observed. If any path from Vi to V22 is blocked, then Vi W.
A.3 Detailed Proof of Theorem
E
K
E
K
X
X
K
X
X
K
Figure Causal graphs for anomaly detection. The left figure shows the case of confounding, The right figure shows the case of confounding. An intervention at the E' variable induces a domain shift (gray hammer), whereas an intervention the X, variable induces a covariate shift (black hammer).
We repeat in Figure the causal graph for anomaly detection here for convenience. To prove Theorem we use the following lemma.
Lemma 2. If W and E are confounded, then X, _ E W. Otherwise, X, L.L. F.
Proof. We prove this via d-separation. We start with the case of W and E not being confounded. Note that there are two paths from X, One via W and the other via Z. The path via W is blocked, because X, is collision and neither X, nor its descendant Z are observed. The path via Z
15

---

is blocked, because Z is collision with no descendants and is not observed. Hence, X., when W and are not confounded. In the case of W and E being confounded, we can show analogously that X, 1 F. W. Note that there are three paths from Xa to F. The first one is via U. but this is blocked, because W in that path, is not a collision, and it is observed. The second one via W and Xe. but W is in that path and it is not a collision there, so that path is also blocked. The third one is via Z, but Z is a collision in that path with no descendants and it is not observed, so is also blocked. Since all paths are blocked, when W is observed, we conclude that X. 1 E' W, when W and E are confounded.
Theorem 2. Suppose that learns invariant representations. If W and E are confounded, then Z PLB E W. Otherwise, Z E.
Proof. Recall that if f learns invariant representations, then we assume it be X..measurable.. This assumption is justified in Veitch et al. [2021]. As result, since Z = =[(Xa Xt), we have that Z is Xa- me-aeauuaalle
We assume that W and E are not confounded and show that Z by proving that P(Z e A, E B) p(Z A)p(E B), for any A. E B (R"). Note that p E A,E B) for some Borel set CA- The last equality fol- lows from Z being -mmmaauuralle, which implies that (A) (CA) for some Borel set CA- by Definition 5 By Lemma 2 we have that Xa E. This implies that (X= (B)) =
(Z~ E (B)) = =
(CA))(CA) p (E- (B)) p (Z- (A)) p (E-10)-00) (B)) p(Z A)p(E E B), which is what wanted to show.
We now assume that W and E are confounded and show that ZLE W by proving that MZEA,EC B = p(( E | W E C)p(E E B W C),
for any A.C E (RR).. By an analogous argument, we can show that (F (C) By Lemma (X, (CA) (O)) W (C).. With arguments similar to those above, we can
P (Z E A, -
7 we have that
show that the last expression is equal to p(Z E A| W C)p(E e B W E C), which is what we wanted show.
B Dataset Details
B.1
Camelyon17
Our realistic anomaly detection dataset derived from the Camelyon17 dataset (Koh et al. (2021), Bandi et al. (018)), and contains 3 x 96 x 96 patches of whole-slide images of lymph node sections sourced from patients who may have metastatic breast cancer. This dataset encompasses tissue patches obtained from five different hospitals. The objective here is to accurately predict the presence of tumor tissue within the patches drawn from hospitals that were not part of the training data. Prior work has shown that differences in staining between hospitals are the primary source of variation in this dataset, however, other divergent factors the sampling distribution include different acquisition protocols and patient populations (Tellez et al. 20199).
The in-distribution data was comprised of 151, 280 images evenly distributed across three hospitals, or 100, 810 images evenly distributed across two hospitals depending on the training setting. The other out-of-distributie second with 85, 054 patches. Note that to adapt this dataset to the anomaly detection setting, only normal images included in the in-distribution training data.
data covered two additional datasets, the first with 34, 904 patches, and the
B.2 Synthetic datasets
The synthetic datasets employed in this study were derived from the ViB-6 benchmark (Eulig et al. (2021). This benchmark uniquely allows for the manipulation of five independent generative factors from colored images: overlaid texture, object size, object position, lightness, and saturation, in addition to the semantic features that correspond to the label. Our synthetic experiments utilized
two datasets: MNIST (Deng 2012)) ) and MaSHI (Xiao al. (2077). All images both
datasets were upsampled dimensions of 3 x 256 x 256.
16

---

Train
Test
id 1
id
id
od
od
od
od
2
1
1
C2
eg
e
4
abnormal normal
4
Increasing cumulative covariate shift
Figure 7: Illustration of our experimental setup for the synthetic covariate shift experiment. The image demonstrates representative examples of training data from two distinct environments, alongside instances of normal and abnormal test data subject to progressively accumulated covariate shifts. This configuration embodies the nuanced challenge identifying subtle, yet potentially consequential, changes in the data distribution.
Initially, we generate two unique and distinct environments specifically designed for the training data. Our primary goal during this stage was to guarantee that all these factors exhibited noticeable differences when compared across two generated environments.
Following the generation of these training environments, we proceeded to develop another pair of environments. These new environments were crafted for the validation data. To ensure consistency, these validation environments were fashioned in such way that they closely mirrored or replicated the factor configuration that was present in the initial training environments, thus retaining an in-distribution setting.
In the final step, six additional environments, denoted eo; 61 .... es, were generated. Each environ- ment e; consists of images in which factors have been altered with respect to eq. For depiction of the samples for these different environments, please refer to Fig. 7
In devising our evaluation setup, we opted for inducing covariate shifts that are minor deviations from the original in nn-distribution environments. This decision was motivated by our goal to simulate subtle yet potentially detrimental covariate shifts, particularly in comparison to the challenge of differentiating normal from abnormal.
description of the accumulated covariate shifts in the test environments, Co, C1, C22 C3, 64 provided in Table
Chosen factor in C₁
0
None
Hue
2
Texture
3
Lightness
Position
Table Environments ....... 6. used in our synthetic benchmark. For 0
the environment
modifies the new factor indicated in the table in addition to the factors modified by 6i-1-
17

---

C Invariantly pretrained encoders
We also extend our experimental evaluation by adding a comparison to Smeu et al. 2022], ]. am environment-awan framework for AD that pretrains the encoder of the AD model using invariance- inducing method (LISA or IRM). We evaluate this method in MNIST and F-MNIST subject to targeted covariate shifts (the exact same setup seen in original experiments). The method was incorporated into all baselines and compared to the same baselines while regularized through partial conditional invariance.
We show the results these experiments in Fig- 8 Across all methods tested, and in both datasets, observe sharp decrease in performance when compared to baseline non non-invariant method and that it provides less robustness to covariate shifts than our proposed methodology.
315
Oruaa 701R
315
00
201R
125
04
04
2016
10
Tou
(a) MNIST
covariate
MNNST
covariate
(c) ea:NNIST three covariate shifts
shift
shifts
Oread
(d) M:NNIST s.t. shifts
four covariate
(e) FNFIST s.t. one covariate shift
(f) e::TSM s.t. two covariate shift
PER
A03000
Bui POOEA
Ied MODEA
(g) ea:F-MNIST s.t. three covariate (h) ea:F-MNIST s.t. four covariate shifts
shifts
Figure 8: Experimental results MNIST and Fashion-MNIST with additional invariant pretraining following. (background transparent bar-plots: in-distribution evaluation; foreground opaque bar-plots: out-of-distribution evaluation). (a-d) Results in MNIST. (e-h) Results in MNSSS.
D Visualization of Invariance vs Informativeness
In Fig. 9 we plot the two-dimensional representation of the final layer of model trained through MeanShift(Reiss and Hoshen| (202T) at different levels of PCIR regularization. The embeddings are obtained through t-SNE. From the progressive increase in the weight of the PCIR term, we see the increased superimposition of the different environments leading to more invariance the loss of
informativeness in the representation.
18

---

4.1
0.0
0.
25
25
-30
Bodie
-30
-3.0
-75
Bieas Lodoo
-25
-75
-25
-5.
-25
00
33
30
-25
-5.
-33
23
30
(a) Invariant and informative
(b) Invariant but non-informative
(c) Informative but non-invariant
Figure 9: TSNE embeddings of MNIST with three background colors for the digits 4 and 9. The model used was MeanShift subject to different degrees of partial conditional invariant regularization. (a) PCIR term 5 (b) PCIR term set 150. (c) PCIR to O.
E
Tables of Results
In dist.
Out of dist.
e1
62 (1) AUROC
63
e4
65
(1) AUROC
(t) AUROC
(1) AUROC
(1) AUROC
STFPM
0.850 0.005
0.883 + 0.014
0.815 0.024
0.753 0.007
0.745 = 0.005
STFPM (PCIR)
0.868 0.007
0.873 = 0.008
0.814 0.014
0.798 0.011
0.774 0.010
ReverseDistil
0.853 + 0.009
0.884 + 0.014
0.797 + 0.015
0.716 0.009
0.723 0.004
ReverseDistil (PCTR)
0.866 + 0.007
0.903 + 0.007
0.857 + 0.010
0.021
0.718 + 0.010
0.010
CFA
0.625 0.002
0.689 + 0.008
0.566 + 0.014
0.454 + 0.005
0.590+0.00
CFA (PCIR)
0.625 0.003
0.700 1 0.007
0.597 0.011
0.473
0.009
0.595 1 0.027
MeanShift
0.751 0.012
0.761 + 0.014 0.014
0.731 1731100000 0.015
0.703
0.009
0.701 1 0.010
MeanShift (PCIR)
0.781 1: 0.014
0.791 0.014
0.778 0.012
0.772
0.770 0.011
CSI
0.601 0.021
0.600
0.582 0.014
+.015
0.531 = 0.014
CSI (PCIR)
0.648 0.022
0.021
0.621 0.020
0.581
0.600 = 0.023
Red PANDA
0.761 0.017
0.764 0.015
0.671 = 0.015
0.641
0.651 = 0.012
Red PANDA (PCIR)
0.761 +: 0.017
0.769 1 0.015
0.742 + 0.019
0.731 0.015
0.740 + 0.015
Table 2: Experimental results on realist domain shift in the Camelyon17 dataset, for both regularized and unregularized models trained on two environments. Results are presented over in-distribution evaluation (environments ej and e2) and over out-of-distribution (environments e3, E4, and E5).
In dist.
Out of dist.
e2 (1) AUROC
C5
(1) AUROC
(t) AUROC
(1) AUROC
(f) AUROC
STFPM
0.854 + 0.011
0.873 + 0.004
0.782 + 0.013
0.771 + 0.013
0.735 + 0.008
STEPM (PCTR)
0.865 0.006
0.873 + 0.00S
0.796 0.014
0.782 0.012
0.759 + 0.009
ReverseDistil
0.830 0.028
0.866 + 0.007
0.781 + 0.004
0.707 0.023
0.710 + 0.033
ReserseDistil (PCIR)
0.843 0.009
0.880 + 0.015
0.799 1 0.004
0.010
0.715 + 0.009
CFA
0.667 1: 0.004
0.708 1 0.011
0,628 1 0.019
0.559 0.005
0.643 1 0.006
CFA (PCIR)
0.667 1: 0.014
0.705
0,623 1 0.004
0.561
0.003
0.644 1: 0.003
McanShift
0.742 0.014
0.701
0.681 0.014
0.739
+
0.690 = 0.013
McanShift (PCIR)
0.772 - 0.013
0.014
0.739 = 0.018
0.731 = 0.013
CSI
0.620 + 0.015
0.014
0.548 + 0.013
0.011
0.541 = 0.015
CSI (PCIR)
0.650 1 0.019
0.650
0,613 + 0.015
0.624 0.014
0.611 1 0.010
Red PANDA
0.751 4 0.012
0.742
0.651 0.010
0.751 4 0.010
0.641 1 0.011
Red PANDA (PCIR)
0.767 0.016
0.781 + 0.014
0.761 + 0.013
0.770 0.012
0.751 + 0.009
Table 3: Experimental results on realist domain shift in the Camelyon17 dataset, for both regularized and unregularized models trained on three environments. Results presented over in-distribution evaluation (environments 61, C22 e3) and over out-of-distribbution (environments c4, and es).
19

---

In dist.
Out of dist.
(1) AUROC
(f) AUROC
STEPM
0.699 0.025
0.630 + 0.025
STFPM (PCIR)
0.724 0.020
0.698 1 0.023
ReverseDistil
0.673 0.057
0.617 1 0.032
ReverseDistil (PCIR)
0.734 + 0.013
0.723 = 0.013
CFA
0.752 0.003
0.705 = 0.018
CFA (PCIR)
0.785
0.759 = 0.005
McanShift
0.683 0.041
0.629 = 0.043
McanShift (PCIR)
0.731 + 0.022
0.726 + 0.052
CSI
0.671 + 0.026
0.626 = 0.024
CSI (PCIR)
0.692 + 0.017
0.674 = 0.019
Red PANDA
0.732 0.026
0.691 = 0.031
Red PANDA (PCIR)
0.742 + 0.029
0.721 = 0.012
Table 4: Experimental results on realist shortcut learning in the Waterbirds dataset, for both regularized and unregularized. Results are presented over in-distribution evaluation and over out-of- distribution.
In dist.
1 cov. shift (1) AUROC
2 cow. shifts (t) AUROC
3 cov. shifts (1) AUROC
4 cov. shifts
(1) AUROC
(t) AUROC
STEPM
0.850 + 0.005
0.883 + 0.014
0.815 + 0.024
0.753 + 0.007
0.745 + 0.005
STFPM (PCIR)
0.868 + 0.007
0.873 + 0.008
0.814 + 0.014
0.798 + 0.011
0.774 + 0.010
ReverseDistil
0.853 0.009
0.884 + 0.014
0.797 + 0.015
0.716 0.009
0.723 + 0.004
ReverseDistil (PCIR)
0.866 0.007
0.903 = 0.007
0.857 = 0.010
0.744
0.718 = 0.010
CFA
0.625 0.002
0.689 = 0.008
0.566 : 0.014
0.005
0.590 = 0.017
CFA (PCIR)
0.625 + 0.003
0.700 = 0.007
0.597 0.011
0
0.595 + 0.027
McanShift
0.751 + 0.012
0.761 0.014
0
0.701 = 0.010
McanShift (PCTR)
0.781 0.014
0.791 0.014
0.778 0.012
0.013
0.770 + 0.011
CSI
0.601 0.021
0.600
0.582 + 0.014
0.491 0.015
0.531 - 0.014
CSI (PCIR)
0.648 0.022
0.652 1 0.021
0.021
0.621 1 0.020
0.581 0.018
0.600 1 0.023
Red PANDA
0.761 1 0.017
0.764 0.015
0.671 1 0.015
0.641 0.015
0.651 1 0.012
Red PANDA (PCIR)
0.761 1 0.017
0.769 1 0.015
0.742 1 0.019
0.731 1 0.015
1 0.015
Table 5: Experimental results on synthetic covariate shift over the MNIST dataset, for both regularized and unregularized models. Results are presented over in-distribution evaluation, and test sets subject
to one four different covariate shifts, as portrayed Fig.
In dist. (1) AUROC
1 cov. shift (1) AUROC
2 cov. shifts (t) AUROC
3 cov. shifts (1) AUROC
4 cov. shifts
(t) AUROC
STFPM
0.752 + 0.010
0.530 + 0.002
0.526 + 0.004
0.526 + 0.004
0,516 + 0.007
STFPM (PCIR)
0.733 + 0.014
0.601 + 0.034
0.566 + 0.004
0.562 + 0.008
0.546 + 0.006
ReverseDistil
0.737 1 0.008
= 1.0130
0,655 = 0.014
0.582 1 0.011
0.543 = 0.014
ReverseDistil (PCIR)
0.740 0.007
0.689 1 0.009
0.683 1 0.008
0.615 2.015
0.564 1 0.004
MeanShift
0.751 0.010
#.009
0.692 +
0.010
0.643 1 0.010
McanShift (PCIR)
0.764 + 0.010
0.74
0.739 0.009
0.010
0.703 = 0.008
CSI
0.691 + 0.016
+.011
0.571 +
0.012
0.542 = 0.014
CSI (PCIR)
0.681 0.014
0.016
0.639 + 0.014
0.611
0.599 + 0.011
Red PANDA
0.729 0.013
0.693
0.013
0.672 0.015
0.637
0.014
0.614 1 0.010
Red PANDA (PCIR)
0.715 0.016
0.690 0.013
0.681 + 0.010
0.661 0.012
0.642 1 0.009
Table 6: Experimental results on synthetic covariate shift over the Fashion-MNIST dataset, for both regularized and unregularized models. Results are presented over in-distribution evaluation, and test
sets subject to one to four different covariate shifts, portrayed in Fig. 7
20

---

Performance Gained Compared to Baseline
Our investigation of the influence of partially conditional regularization on model performance further expanded in Fig This figure presents the percentile difference mean-AUROC (Area Under the Receiver Operating Characteristic) between partially conditionally regularized and unregularized models.
By comparing each regularized model to its unregularized equivalent across all the tested environ- ments, have been able observe a consistent improvement performance when regularization is applied. This observation holds true across all models and out-of -iit-iiiittiittion environments studied.
The increase in performance due to regularization varies from 2.5% substantial 20% in some models. This variability implies model--peeific dependency on the degree of invariance that can be induced. Crucially, however, regularization substantially bolsters -it-of--iitributiin performance without any additional training cost, reinforcing its merit as an effective strategy for developing robust anomaly detectors.
To further highlight this point and for the sack of completeness, we also present in Fig. Fig and Fig. 13each regularized model compared to its baseline.
<7
(b) Camelyon distribution
three in-
(a) Camelyon17: two in in-distribution
AAOC dkewn [N)
(c) MNIST
(d) Fashion-MNIST
Figure 10: Percentual performance gain over each regularized model when compared unreglarized baseline.
G Ablation Studies
As part of our comprehensive examination of how covariate shifts influence individual regularization weights, we have plotted the performance trajectories of all evaluated models, traversing environments fromeg e4. These are captured in Fig 14 and Fig 15
As expected, and already observed both our main findings and previous work (Ming et al. [022 review of these plots unveils a trend that permeates across all examined models: the performance of models appears to inversely correlate with the number induced covariate shifts. As the com- plexity introduced by these shifts mounts, the models' performance experiences proportionate and systematic decline. This observable trend is essentially monotonic, signifying a erosion in model performance with each incremental rise in the quantity of covariate shifts.
However, it important to note that this trend not without exceptions. In particular, when scrutinizing the data pertaining to environment 64, we can observe anomalies to this downward trend. In these exceptional instances, despite the increase the number of covariate shifts, the performance of certain models appears resist the general declining pattern.
21

---

STAPM
ReveseBent
090
000
080
100
04.
(a) Camelyon17: STEPM
(b) Camelyon17. ReverseDistil
(c) Camelyon17: CFA
Rud PANDA
10:0
622
6.00
6300
(d) Camelyon17: McanShift
(c) Camelyon17: CSI
(f) Camelyon17: Red PANDA
Figure 11: Mean-AUROC curve of each anomaly detector and its regularized version the Cam- 1 dataset.
STAPM
RestseDo
002
Fereerenh
Feceeranh
Faueerenh
(a) MNIST: STEPM
(b) MNIST: ReverseDistil
(c)
MNIST: MeanShift
69
Rel PANDA
307
8:34
(d) MNIST: CSI
(e) MNIST: Red PANDA
Figure 12: Mean-AUROC curve of each anomaly detector and its regularized version in the MNIST dataset.
Thus, our overall conclusion, while acknowledging these exceptions, is that the prevalence of covariate shifts largely contributes to a degradation in model performance.
It is however important to note that the unregularized methods still underperforms when compared to the same method under a even small amount (0.001) of partially conditional regularization added.
H
Additional Discussion
H.1 Shortcut Learning in Anomaly Detection
To expand on the experiment tackling real-world shorteut learning, and to better understand how a
distribution shift affects different kinds of shortcut features (Geirhos et al. (2200) captured by the model, we now will look at how inducing distinct changes to the anomaly detection causal graph may lead to malfunctions in the model.
22

---

STAPM
RevereDiot
000
022
000
Favkwwech
Faveerachh
(a) Bushion-MNISTT STFPM
(b) -NNiiIT ReverseDistil
(c) Fashion-MNIST: MeanShift
Red PANDA
8.56
150
1.2
Sokee
(d) Fashion- CSI
(c) Bushion-MNISTT Red PANDA
Figure 13: Mean-AUROC curve of each anomaly detector and its regularized version in the Fashion- MNIST dataset.
ST:PM
Mex
0.00
(a) MNIST: STFPM
(b) MNIST: Revers
(c) MNIST: MeanShift
Fad PAND6
k00
8.00
(d) MNIST: CSI
(c) MNIST:Red PANDA
Figure 14: Ablation study over the weight of the regularization term for MNIST under distribution shift, with separate plots for each model.
Suppose we recover our formulation of the partition of an object X into the semantic features that distinguish normal and abnormal samples, X,,, and into the style features induced by the environment, X.. In that case, it is possible to distinguish between settings that may lead a model failure when shortcut feature is captured.
Let us simplify our analysis by considering the setting where the training data is sampled under the intervention (o((=ee)( that is the style features are fixed into specific setting, X". This is a prevalent setting in real-world applications as spurious correlations between style and semantic features may occur when sampling the training data. Remember that in the training set, Xa only produces features of normal objects. Under this constraint, it was already previously noted that anomaly detection methods are particularly susceptible capturing the style features as prominent factor for the representation of X (Ming et al. [2022)) )
Moving to the evaluation stage of the anomaly detector, we can then consider two settings.
23

---

STAPM
ReveseBicl
1.0
6030
1000
000
Farswres
Far
(a) Bushion-MNISTT STEPM
(b) Bushion-MMISSS Reverse Distil
(c) Fashion-MNIST: MeanShift
107
8.2
3.04
(d) Fashion-MNIST: CSI
(e) Fashion-MNIST: Red PANDA
Figure 15: Ablation study over the weight of the regularization term for Fashion-MNIST under distribution shift, with separate plots for each model.
The first setting consists of no changes to the intervention p that is, there was no distribution shift in the sampling of the data. In this setting, the main surrogate for model failure is derived from anomalies that are characterized by small changes in X, from normal to abnormal samples. That is, the style features would be considered the main source information to characterize new samples, and under these constraints, it is only natural that all test instances would be highly likely to be set as normal. This setting was introduced by both Ming et al. [2022] and Cohen et al. 2023), and the underlying features can be referred to nuisance features.
The second setting consists of a different intervention, p glo("==e)) (X) (X) such that it differs from the training density zdo(E==') (X). In particular, we can consider an intervention that only changes stylistic features, Xe, captured by the model. We are essentially operating under a highly targeted covariate shift that focuses on the shortcut features. Therefore, depending on the extent of the changes in X2 from normal to abnormal samples and how well they are captured by the model, this distribution shift would lead to anomaly detector that classifies all new samples as anomalies.
Note that both settings, as shown our main presentation of the method, by inducing a partially conditional invariance to different environments, our regularization method also inherently introduces an invariance to the style features Xe. As supported by our results in the synthetic covariate shift experiments, we believe this also produces models that not only robust to covariate shifts, in the second scenario, but also to shortcut learning, as described in the first scenario.
H.2 Data Augmentation
Although data augmentation is well-established technique tackle model robustness to distribution shifts, there are two main problems with solely relying on data augmentation to solve problem of distribution shift. First, data augmentation relies both knowing of the existence of a distribution shift, and a way to simulate it. This could be achievable when the distribution shift is characterized by transformations easy to simulate through data transformations. However, in real-world settings, this is rarely the case: distribution shifts are complex transformations that are almost impossible identify, and in many cases impossible to programmatically simulate. For example, in the case of the Camelyon17 dataset that considers histological cuts from different hospitals, the changes between environments that are derived from the change in histological staining are easy to visually inspect, and could even be simulated. But, by changing the hospital, the patient population also changes, and with several cofounders, such as group age patient comorbidities. These are both impossible to know without further annotation and unfeasible to simulate. Furthermore, data augmentation does not induce an invariance to particular distribution shift, giving additional "guarantees" It only increases the
easily identifiable attributes (c.g. the color of digit), and which are also
24

---

pool of examples of the data in hopes that the model implicitly captures the additional simulated variance in the images.
Yet, data augmentation could still be beneficial the context of penalized invariant regularization, by producing meaningful augmentations in the context of multi-environment setting. It could be used to generate additional data for single intervention, thus increasing the pool of available samples of a specific environment, be used generate new data for new intervention, leading to an increased number of environments available, and the overall pool of samples in the dataset. This would effectively alleviate the second drawback of solely using data augmentations.
H.3
Choice of Distance Metric
We considered other options aside from MMD, like the Wasserstein distance. MMD known be computationally easier than Wasserstein's 's distance. This is because MMD can be easily computed using Gaussian kernels, whereas the Wasserstein distance requires computing supremum over of probability measures, which is computationally intractable in general.
Another option considered the KL-divergence. However, previous works have demonstrated that this divergence is very unstable for probability distributions supported on low-dimensional manifolds (Arjovsky al. (017]). Note that many of the methods used for anomaly detection and machine learning are trained on samples from such distributions. A similar argument applies for variations of this divergence like Shannon-Jensen divergence and total variation.
Finally, we remark that MMD has a strong theoretical foundation. In spite of its simplicity, it derived from supremum of differences of expected values of functions from reprodacible-kerne Hilbert space (Gretton et al. (0122). It has been shown that when this metric is 0 then the two distributions must match, which is precisely what aim to achieve when learning representations that are invariant to the environments.
I Implementation Details
I.1 Baselines
STEPM The STFPM (Wang et al. [202]) algorithm incorporates a pretrained teacher network and student network that share the same structure. The student network assimilates the distribution of images devoid of anomalies by aligning the features with corresponding features in the teacher network. To boost robustness, the algorithm uses multi-scale feature matching. This multi-tiered feature alignment lets the student network absorb a blend multi-level insights from the feature pyramid, thereby permitting the detection of anomalies of varying magnitudes.
During the inference stage, the feature pyramids of both teacher and student networks are put into comparison. A larger discrepancy between these pyramids implies a heightened likelihood of the presence of anomalies.
Reverse distillation The reverse distillation method (Deng and [022) is assembled from three networks: an initial pretrained feature extractor, f. a bottleneck embedding, ó. and the student decoder network, p. The primary layer, or the backbone, of f derived from a ResNer model that was pretrained on the ImageNet dataset.
During the execution of forward pass, the model extracts features from three separate ResNet blocks. These features are encoded by amalgamating the three feature maps using the multi-scale feature fusion block of Φ, and then transferred the decoder, v, which is constructed mirror the feature extractor, albeit with operations reversed.
Throughout the training process, the output of these mirrored blocks is made to match the outputs from the respective layers the feature extractor. This is ensured by adopting the cosine distance as the loss metric.
CFA Feature Adaptation based on CFA eLe et al. 2022 identifies anomalies utilizing features that are specifically tailored the target dataset. The CFA model comprises two main elements: firstly, learnable patch descriptor that leams and assimilates features oriented towards the target,
25

---

and secondly, scalable memory bank that remains unimpacted by the size of the target dataset. In conjunction with pre-trained encoder, CFA applies a patch descriptor and memory bank. By doing so, it makes use of transfer learning to bolster the density of normal features. Consequently, this facilitates an easier distinction between normal and abnormal features.
Mean-shifted The mean-shifted contrastive learning method (Reiss and Hoshen 2021]) introduces a novel loss function that calculates angular distances using the mean of all feature vectors as a reference point. This is done in contrast to using the origin as a reference and the Euclidean distance. It also combines two loss functions, one involving contrastive terms akin to Chen et al. [2020), but these terms are positioned around a hypersphere centred on the mean of all feature vectors. To deter positive samples from repelling themselves, it also incorporates an angular centre loss that encourages samples to gravitate towards the mean of normal samples.
CSI CSI (Tack et al. 2020 is a direct extension of Chen et al. 2020 introducing a unique form of data augmentations known as distribution-ssifting augmentations. In this setup, distribution-shifeed augmentations are treated negative samples instead of positive ones and are consequently pushed away from all positive samples. These augmentations include manipulations such as rotations and permutations. The augmentation's potential shift the distribution is assessed through the AUROC, where samples altered by the said augmentation are considered out-of distriibbiii samples. The underlying notion here is that distinguishability directly proportional to the shift in distribution.
The Red PANDA method for anomaly detection Cohen et al. 2023 tackles the particular problem of anomaly detection under nuisance or distracting features. Relying on labels from the nuisance factors, it employs contrastive disentanglement loss following Kahana and Hoshen [022], in conjunction with a perceptual loss to train a generator function end-to-end with a pretrained encoder.
Red PANDA
1.2 Anomaly Scoring
STEPM During training, the student feature tries to align the distribution of training dataset with the teacher. During prediction, the input x is fed into both student and teacher feature extractors, where the student outputs (1) and teacher outputs fz).. For the anomaly scoring, it relies in a traditional density estimation approach. The assumption would be that the normal samples are mapped to the high density area where the student encoder and the teacher encoder are aligned, and the anomalous samples are mapped to the low density area of the student extractor. Therefore, the anomaly score is computed as the distance between f1) and fr(x), i.e. AS(x) = d(f.(x), fr())) In this case the distance metric d is the 12-norm.
Reverse distillation The anomaly scoring function here is derived from the standard anomaly scoring functions used in econstruction-basee anomaly detection algorithm. In particular, the anomaly score is defined as the distance between the encoded features and the reconstructed features from the decoder. AS(s) d(f (v), (f)(f())) where v(.) is the decoder, Φ(-) is the distiller and f(-) is the encoder. The idea behind this anomaly scoring function that the decoder has learned to reconstruct normal samples due training dataset, but isn't 't able to reconstruct anomalous samples. Therefore, anomalous samples would have a higher anomaly score.
CFA For CFA we use the standard imaee-level density estimation approach. In particular, the training samples are mapped to a feature map f(x) and clustered into & clusters using k-means. For the prediction, given an input y, its final feature / (yy) is computed after feeding it into the feature extractor and descriptor. Then the d nearest cluster centers of f()) would be selected, and the anomaly score for that sample is the mean distance to those d centers. AS(y) = E,eN,, (x)) (f f(y). ). In this case, the distance metric d is simply the l2 distance. The idea behind this approach is to assume that the anomalous samples would be mapped to low-density area in feature space, which are far away from all the cluster centers, while the normal samples would be mapped to high density area in feature space, which is close to the normal samples in training dataset.
Mean-shift and Red PANDA Both contrastive learning based methods used as baselines, namely, mean-shift (Reiss and Hoshen (021] and Red Panda (Cohen et al. (2023) ) rely finetuning an
26

---

encoder (both pre-trained or not) by grouping the set of feature vectors from images in the training data around a sub-region of the hypersphere centered in the origin. During prediction time, the most common approach to classify new sample as anomaly not is through the mean distance of the KNN normal images. Following the original work, in mean-shift, k was set to 2, and in Red Panda was set to 1.
highly augmented samples to relied on only a vanilla contrastive loss between original samples, and proxy for abnormal samples. Similarly to the previous setting of mean-shiff and Red Panda, this leads to a feature space that falls around the hypersphere centered in the origin, but not necessarily in the surface. However, operating with the underlying hypothesis that the highly augmented samples match the anomalies, the feature vectors from images in the training data are already being pushed to the diametrically opposite side of the hypersphere when compared to abnormal samples. Additionally, it was empirically verified that the norm of vectors of abnormal samples is much lower than that of in-distribution samples. This leads to a distance criterion that measures the closest training sample through the cosine similarity and the norm of the feature vector of the sample: maxm sim(f (mm)). (x f=)) [x]],, where is the encoder that maps the input object, to its feature space, and sim is the cosine similarity.
CSI CSI (Tack et al. 2020)
1.3 Hyperparameters
As this work considered a novel setting where each anomaly detection method was evaluated for the first time, we modestly optimized hyperparameters. Our approach consisted of two primary steps. The first involved scaling up two key factors: (a) batch size, and (b) learning rate. Subsequently, we methodically scanned through an array of distinct parameters for each baseline model. These included the backbones ResNet18, ResNet34, ResNer50 and WideResNet50, alongside various anomaly scoring methodologies that leverage image-level, density estimation, reconstruction error, and pixel-wise density estimation approaches. An additional aspect of our study analysis where the regularization weight was fine-tuned by sweeping through the set of values 0.001, 0.01, 0.1, 1, 10, 100. We adhered to the hyperparameters depicted in the original works for all other variables and refrained from performing any further optimizations on them.
ablation
1.4 Backbone choice
One thing we notice during our experiments is that for models that rely on the pretrained backbones, the choice of backbone matters. For instance, for STFPM, the optimal choice was the simplest feature extractor ResNet18, 18, but for reverse distillation, the optimal choice was WideResNet50.. The choices seem to be model dependent more than dataset relevant. For details the chosen backbone for each method refer to Tab. 7
1.5 Computational resources
The complete project required 3400 hours of GPU usage throughout all experiments, covering development, testing, and comparisons. The resources supplied were part of local custer, and consited of two GPU models: the NVIDIA TITAN RTX and the NVIDIA Tesla V100.
I.6 Code and Licensing
The main Python libraries used in our implementation, were Pytorch, which is under a BSD-3 license and Pytorch Lightning, which is under Apache 2.0 license
Methods that were derived from the anomalib library (Akcay et al. 2022)). ), namely STFPM, reverse distillation, and CFA, were already implemented as Pytorch Lightning Module, and are all under an Apache 2.0 Ficense These were incorporated directly in our pipeline.
P hhupp://gttuub.cmm/.gggiiiii http::/www....oogliggnngnngnnnnn wtupr:/..hhcooooooooooooooooomod
27

---

STEPM
ReverseDiuil
CRA
McunShift Retes and Toshen
CSI
Red PANDA
Wang al 2021
Deng and
2022
Lex et al
2022
2021
Tack et al
2020
Cuhen al.
2023
Camelyon 17 (3x96x96)
learning rate optimizer beach size backbone pretaining best reg. weight
10-2 SGD
10-2 Adam WideResNet50 True
10-3 Adom
10-a Adam
10-3 Adam
10-4
SGD
128
ResNet18
ResNet18 True
ResNet50 True
ResNet18 False
ResNet50 True 10 density estimation
True
100 density estimation
unomaly
reconstruction errer
density
density estimation
donsity estimation
eslimation
V (MNIST) (3x256x256)
learning rate
10-1
10-2 Adam
10-3 3 Adam ResNet50
10-3 Adam
3 10 SGD 32
optimizer batch
SGD
32 ResNet18
backbone
WideResnet50
ResNet18 False
ResNNe50 Truee 10 density estimation
pretaining
best reg. weight
10
anomaly score
density estimation
reconsrruction errer
density estimation
density estimation
Ding (Fashion-MNiST) (3x256x256)
learning rate
10-1
10-2
10"h Adam
10-3 32 ResNet18 False
3 10 SGD 32 ResNet50 'True
optimizer batch size
SGD
32
32
32 ResNet50 True
backbone pretaining
ResNer18 True
WideRestNet50 True
best reg. weight
0.1
anomaly score
density
reconstruction
density
deusity
density
estimation
estimation
estimation
estimation
Table 7: A detailed summary of the hyperparameters used for each evaluated model across three datasets: Camelyon17, DiagVit V (MNIST), and ViagV
Parameters include learning rate, scheduler, optimizer, batch size, backbone, pretraining, regularization weight, and mmd kernel type, along with the type of anomaly score. Notably, the CFA model could be successfully implemented for Diag based experiments despite trying an extensive range of hyperparameter combinations. Models are referenced by their respective citations.
DCoDR (Kahana and Hoshen 2022 ). which Red Panda is based from, was released under
Software Research License"] Our experiments for Red Panda were derived directly from the official repository. Mean-shifted contrastive learning was released under a Software Research License We
re-implemented this method Pytorch Lightning Module, loosely following its original official
implementation. ViB-6 (Eulig et al. [202T) and the Camelyon17 (Koh et al. (202T) datasets were also publicly released with a GNU Affero General Public License v3.0 and MIT License respectively. Our implementation follows directly from its official repository.
28


Reviews:
[{"review_id": "9qrQ6zEJVi", "reviewer": "Reviewer_zVDG", "summary": "The paper proposes a formalism for detecting anomalies in the presence of domain and covariate shifts. Data X is assumed of being decomposable into X_a and X_e where the latent factors X_a determine if X is an anomaly while X_e represents style features. To obtain invariant features, a regulariser is proposed. It is based on MMD between the features of samples from different environments, conditioned that they represent normal samples. Experiments on synthetic and real data show that the proposed regulariser improves the performance of multiple anomaly detection methods.", "questions": "See Weaknesses.", "limitations": "There is no negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and the formalization is sound. \n \n 2. The MMD regularisation is appropriate for the given graph assumptions.\n \n 3. Experiments show good results in multiple settings.", "weaknesses": "1. A similar framework for anomaly detection in the presence of style changes is proposed in [A]. They also aim to solve anomaly detection based on content features (similar to X_a) while being invariant to style features (similar to X_e). Similarities and differences between the frameworks should be discussed.\n\n2. Conditional MMD methods have been applied before for domain generalization [B,C]. This is completely fine, is shown that the proposed baseline is appropriate learning invariant features. The authors should discuss the relation to previous works in domain generalization and highlight the importance of partial conditioning. \n\n3. It is not clear that partial conditioning on W = 0 (apply MMD only on normal data) is enough for obtaining invariant representations in the confounded case. It doesn’t seem the case. The paper should have a discussion about this aspect.\nIt is noted that “domain shifts are stronger than covariate shifts” but this is not formalized anywhere nor is it considered in the method. How important is this fact for the overall framework?\n\n4. Some clarifications should be made about the following: The description in the background section (3.1) does not seem to match the causal graphs in Figure 2. “Conceptually, X_a represents the component of X that determines whether X is an anomaly, while X_e comprises style features. These style features, while unaffected by the anomaly status of an object, are influenced by the environment.” This description would suggest that the anomaly status (W) will affect only X_a but not X_e, but we have an arrow between W and X_e in Figure 2. Can the authors clarify this?\n\n\n[A] Smeu et al. \"Env-aware anomaly detection: Ignore style changes, stay true to content!.\" arXiv (2022)\n[B] Long et al. “Transfer feature learning with joint distribution adaptation,” ICCV (2013)\n[C] Kang et al. “Contrastive adaptation network for unsupervised domain adaptation,” CVPR (2019)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation of the paper, and their comments and suggestions for improvement.\n\n**1. A similar framework for anomaly detection in the presence of style changes is proposed in [Smeu et al. 2022].**\n\nWe thank the reviewer for identifying this framework. Although this work also proposes an environment-based approach for robust AD, no connection is made to covariate or domain shifts, nor to any causal inference study in AD. It proposes a two-step method: first, an encoder is pretrained using an environment-based approach (e.g. IRM) in an unrelated pretext task using the same data, second, an AD method is trained on top of the encoder. This both increases the cost of training such a model and deprives the possibility of using strong pre-trained models, which SOTA AD are highly reliant on. To evaluate this framework we have incorporated it into the baselines (state-of-the-art DL-based AD methods) and evaluated its performance in the MNIST and Fashion MNIST under covariate shifts. Our experiments demonstrate that the approach both underperforms when compared to a baseline non-invariant method and provides less robustness to covariate shifts than our proposed partial conditional invariant regularization. For more details please refer to the global response, point 2.\n\n**Changes**: We will add these results to our experimental section and will update our related work section accordingly.\n\n**2. The authors should discuss the relation to previous works in domain generalization and highlight the importance of partial conditioning.**\n\nWe agree with the reviewer that the additional discussion and references provide increased validation for our approach.\n\n**Changes:**\nWe have therefore extended the introduction with the following sentences :\n*\"Similiar approaches relying on MMD regularization for domain generalization in different tasks have proven to be valuable [B, C]. However, to our knowledge, all methods relying on conditional invariance have been applied under the assumption of access to all classes during training. This is not the case in the AD setting.\"* \n\nWe also updated line 212  in section 4.1 as follows:\n *“We call this approach partial conditional invariant regularization, as it induces conditional distribution invariance over only one instantiation of W, contrary to other MMD based approaches [B,C]”, which have access to all instances of W. “*\n\n\n**3. (A) It is not clear that partial conditioning on W = 0 (apply MMD only on normal data) is enough for obtaining invariant representations in the confounded case.**\nPlease refer to the global response, point 3.\n\n**Changes:** We add a sub-section to the discussion where we address to limitation of partial invariance conditioning in AD, and where summarize the insights presented in the global response.\n\n**3. (B) It is noted that “domain shifts are stronger than covariate shifts” but this is not formalized anywhere nor is it considered in the method. How important is this fact for the overall framework?**\nThis was an observation mentioned in a previous work [R. Geirhos et. al. (2020) \"Shortcut learning in deep neural networks.\"] and we added it for completeness. We agree that in our context this is not relevant and we can remove it in the next version of the paper.\n\n\n**4. The description in the background section (3.1) does not seem to match the causal graphs in Figure 2. “Conceptually, X_a represents the component of X that determines whether X is an anomaly, while X_e comprises style features. These style features, while unaffected by the anomaly status of an object, are influenced by the environment.” This description would suggest that the anomaly status (W) will affect only X_a but not X_e, but we have an arrow between W and X_e in Figure 2. Can the authors clarify this?**\n\nThe reviewer is right. The sentence “while unaffected by the anomaly status of an object, are influenced by the environment.” is incorrect. The style features are also influenced by the environment, as Figure 2 shows. \n\n**Changes:** We will update this sentence as follows: *\"Conceptually, X_a represents the component of X that determines whether X is an anomaly, while X_e comprises style features.\"*\n\n\n"}, {"review_id": "I5rnzLEKCu", "reviewer": "Reviewer_ePdN", "summary": "The paper tackles the problem of anomaly detection under distribution shifts from the perspective of causal inference. Causal inference aims to induce invariance across environments. The authors derive a regularization term for anomaly detection to achieve this goal. The resulting detector is robust to domain and covariate shifts, improving the detection performance under specific distribution shifts. Experiments are conducted on one real medical image datasets involving domain shifts and one synthetic dataset based on transforming MNIST.", "questions": "Here is a mixed list of questions and suggestions:\n* It seems Def. 3 (mutual information) and Section 3.4 are kind of redundant.\n* Why not put variable $Z$ into Fig. 2? In this way, the independence assumption can be readily read off from the figure.\n* In Figure 3 and Figure 4, which bar corresponds to in-distribution performance and which one corresponds to out-of-distribution performance?", "limitations": "The method requires multiple-domain training data, which could be infeasible under some circumstances.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Authors experiment on a real-world dataset Camelyon17, which composes of distribution shifts, to demonstrate the effectiveness of their method. \n* I like the way of motivating the problem (Fig. 1). \n* As far as I know, this work is the first to apply causal inference techniques to anomaly detection tasks (which could have many citations). However, as a work that sets up the stage, the authors can do more rather than mere applications (see Weakness).", "weaknesses": "As the first work that tries to set up the stage for invariant anomaly detection with causal inference, I think there are still some todo's.\n* There is a lack of justification of using the proposed solution rather than other possible causal learning techniques, e.g., invariant risk minimization or other methods that avoid shortcut learning. Unaddressed justification questions like: What is the uniqueness of the proposed solution compared to other causal learning methods? Why do you choose MMD instead of other discrepancy metrics? What are the potential adverse consequences of conditioning on only normal samples (in the case of W=0) and why do we not need to worry about? etc.\n* I'm not convinced about the definitions of covariate shifts and domain shifts. In the current formulation, I don't see the difference between Def. 1 and Def. 2 because line 117 states: \"the environment E only influences $X_e$\". A clarification or concrete examples to illustrate the definitions could be helpful.\n* Experiments are kind of similar and lack variability to demonstrate the significance of the proposed method. The environment only change the number of shifts at training time or at test time. There are some known image datasets in shortcut learning literature (waterbirds and CelebA) and OOD detection literature (CIFAR10-C). Maybe showcasing the proposed method in those datasets as well is more persuasive as they are dedicated to this setup and also real word natural images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and constructive feedback.\n\n**1. (A) There is a lack of justification of using the proposed solution.**\nWe agree with the reviewer that our work would benefit from a more comprehensive background of other causal methods. \n\nWe verified that the state-of-the-art in AD is based on deep representation learning, which sets the stage for invariant representation learning methods. Other techniques like invariant risk minimization or FISH [A] could not be adequate, as they are optimization techniques that were not designed for learning invariant representations. Furthermore, as we have pointed out in Section 2.2 (lines 104-105), it has been shown that IRM underperforms in real-world settings.\n\nFor other methods, such as LISA [B], we see that they only tackle specific types of distribution shifts through selective data augmentation, and therefore require an understanding of the type of shift that has been applied to the data.  This can be prohibitive in many realistic settings.\n\n**Changes:**\nFor our next version, we have produced the following concrete changes. First, we summarize the discussion above and incorporate it into our introduction while motivating. With this, the reader can see why we formalize anomaly detection using causality theory and choose to use follow an invariant representation learning approach. In addition, this shows why other causal learning techniques like invariant risk minimization or methods against shortcut learning are not enough to solve this problem. Second, we expand our related work section 2.2 by explaining why other shortcut learning-avoiding methods are not suited for the constraints of anomaly detection in the presence of domain shifts.\n\n[A] Shi, Yuge, et al. \"Gradient matching for domain generalization.\" ICLR 2022.\n[B] Yao, Huaxiu, et al. \"Improving out-of-distribution robustness via selective augmentation.\" ICML 2022\n\n**1. (B) Why MMD instead of other discrepancy metrics?**\nWe considered other options aside from MMD, like the Wasserstein distance. MMD is known to be computationally easier than Wasserstein’s distance. This is because MMD can be easily computed using Gaussian kernels, whereas the Wasserstein distance requires computing a supremum over a set of probability measures, which is computationally intractable in general. \n\nAnother option we considered was the KL-divergence. However, previous works have demonstrated that this divergence is very unstable for probability distributions supported on low-dimensional manifolds [C]. Note that many of the methods used for anomaly detection and machine learning are trained on samples from such distributions. A similar argument applies for variations of this divergence like Shannon-Jensen divergence and total variation.\n\nFinally, we remark that MMD has a strong theoretical foundation. In spite of its simplicity, it is derived from a supremum of differences of expected values of functions from a reproducible-kernel Hilbert space [D]. It has been shown that when this metric is 0 then the two distributions must match, which is precisely what we aim to achieve when learning representations that are invariant to the environments. \n\n**Changes:**\nFollowing this discussion, we have added a section to our Appendix motivating the use of MMD where we highlight the reasoning behind our choice. \n[C] Arjovsky, Martin et al \"Wasserstein generative adversarial networks.\"ICML 2017\n[D] Gretton, Arthur, et al. \"A kernel two-sample test.\" JMLR 2012\n\n**1. (C) Consequences of conditioning on only normal samples**\nPlease see the global answer. \n\n**2. In the current formulation, I don't see the difference between Def. 1 and Def. 2.**\nIndeed, environment E only influences X_e, but interventions on X_e may be different from interventions on E. For example, consider intervening only on certain pixels of an image. This would be an intervention on X_e, but not an intervention on E, as the environment has not been modified to another valid environment. This shows also that interventions on E are stronger than interventions on X_e. For this reason, we focus on this work on methods that can detect anomalies even in the presence of these stronger interventions. \n\n**Changes**\nWe added this remark to Section 3.3. \n\n**3. Experiments are kind of similar and lack variability to demonstrate the significance of the method.**\nFollowing the reviewer's suggestion, we have added additional experiments in Waterbirds. See global answer for more details.\n\nWe would also like to point out that the synthetic covariate shift experiments not only change the number of shifts at training and test time but also evaluates the model with unseen style instantiation, as described in Section 5.1, lines 236-241. For example, the model is trained with red and purple digits positioned in the upper left and right corner of the image, and evaluated with yellow digits in the lower central position. Furthermore, the experiments in Camelyon17 serve a distinct purpose of evaluating domain generalization under a real-world medical imaging setting, with unknown environment interventions.\n\n**Further comments:**\n- We will summarize Section 3.4 and merge it into the previous section.\n- We will add Z to Figure 2\n- In all figures, the in-distribution setting corresponds to the transparent bar plot, and the out-of-distribution corresponds to the fully opaque bar plot. We provided this information for Figure 4, but not for Figure 3. To make it clearer, we will update both figures with the caption of the Waterbirds experiment.  \n\n**Limitation**\nWe would like to point out that the referenced limitation is acknowledged in our discussion (2nd paragraph of Section 6), where we provide possible solutions. These include relying on additional meta-data, which in many real-world settings is available, to overcome non-existence of domain labels [E].\n[E] Y. Lin et al. \"When and how to learn invariance without environment partition?\" NeurIPS 2022"}, {"review_id": "xdsCELgBCs", "reviewer": "Reviewer_ZicU", "summary": "This paper aims to promote AD through the lens of causality. To this end, the authors propose to minimize the distribution discrepancy of the leaned representations among environments. Some experiments are conducted to verify the effectiveness of the proposed method.", "questions": "cf Weaknesses", "limitations": "cf Weaknesses", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 Exploring AD through a causal perspective is promising. \n\n2 One of the main contribution of this work is the empirical observation, where the authors show that existing AD method may fail when meeting distribution shifts. Meanwhile, the experimental results are solid and the performance gain is decent.", "weaknesses": "1 The motivation figure is vivid, while it is built upon synthetic data. I suggest the authors to give a corresponding figure using features learned with the proposed method, which may be a direct support to the efficacy of the proposed method.\n\n2 The definitions used in this work is confusing, where the authors use both (content, style) and (related, style) notation. A clear definition is required.\n\n3 Corollary 1 is confusing. The authors claim Z=f(X), however, f is typically trained with the data X. Consequently, the independent mechanism assumption does not hold any more. Thus, Z should not appear in Corollary 1.\n\n4 The novelty of the proposed method is limited, where the authors merely control the MMD for two distributions with different representations, i.e., varying with environments.\n\nTypos:\nLine 131, f1, f1, and f1 should be f1, f2, and f3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions to improve this work. \n\n**1. I suggest the authors to give a corresponding figure using features learned with the proposed method.**\nWe agree with the reviewer's suggestion and have produced a TSNE plot of representations produced with different levels of regularization. \nSuch a figure is presented in Figure 3 of the .pdf attached to the global answer. \n\n**Changes:**\nWe will add this figure to our manuscript.\n\n**2. The definitions used in this work is confusing, where the authors use both (content, style) and (related, style) notation. A clear definition is required.**\nWe have tried to find in our work the usage of \"related\" notation, but failed to do so. Could the reviewer point us to a specific example where it was used. We would appreciate the chance to clarify any confusion in our notation. \n\n**3. Corollary 1 is confusing. The authors claim Z=f(X), however, f is typically trained with the data X. Consequently, the independent mechanism assumption does not hold any more. Thus, Z should not appear in Corollary 1.**\n\nWe thank the reviewer for carefully reviewing our proof.\n\nThis is correct: $f$ is trained with $X$, so $Z$ is, in general, not independent of $X$. However, we assume that f learns invariant representations. Under this assumption, we demonstrate Corollary 1. We exploit the insight that f must be $X_a$-measurable, which means that $X_e$ does not influence the outcome of $f$. This observation allows us to demonstrate via d-separation that $Z$ is independent from $E$, when f learns invariant representations. \n\n**Changes:**\nWe changed the proof of Corollary 1 to highlight that $f$ is trained with data $X$. We have also added to the Appendix a more detailed proof. In particular, we explain why all paths from $Z$ to $X_e$ are blocked and added a recap on d-separation and confounding.\n\n**4. The novelty of the proposed method is limited, where the authors merely control the MMD for two distributions with different representations.**\n\nIn this section, we would like to expand on the novelty and value of the methodological contribution and would like to make the case that our method can actually be instrumental within the context of AD subject to distribution shifts.\n\nIn particular, matching the two distributions from different environments is a essential condition for generating invariance towards, for example, a covariate shift. Solving this problem would then be akin to solving domain generalization under this induced shift. (if the invariance across distributions translates to new domains). To our best understanding, this is still an unresolved problem in the field of machine learning, and the most recent attempts have been led by causal approaches (as we describe in our related work in Section 2.2.) Furthermore, approaching a invariant representation learning using conditional distribution invariance is non-trivial, due to the nature of AD only revealing normal samples during training. \n\nMMD has been used before to regularize the training of models, but what makes this insight valuable is that MMD itself is not enough. It requires the conditioning on W = 0.  Likewise, we believe that having a solid theoretical grounding for our method and using causality is a much better approach as it not only provides theoretical foundations (c.f. Corollary 1), but also enhances our understanding of the problem and opens the door for further improvements.\n\nA final important observation is that in spite of being a conditional MMD, and in spite of being only a necessary condition, it is surprisingly strong enough to greatly improve the state-of-the-art anomaly detectors in the presence of domain shifts.\n\nWe see the use of MMD as only one of the results of our causal analysis on anomaly detection. Our analysis has additionally provided the insights above on anomaly detection in the presence of distribution shifts, setting the stage for additional advancements relying on causal inference tools.\n\n\n"}, {"review_id": "WoU3Avi3X2", "reviewer": "Reviewer_yiE8", "summary": "The paper focuses on the making anomaly detection models robust to various kinds of distribution shifts over normal/benign samples. This goal is achieved by formalizing the anomaly detection problem from the information theory perspective and introducing the requirements it brings for developing AD models that are robust to distribution shifts. Based on this observation, the paper proposes a regularization term that ensures the partial distribution invariance across environments. Lastly, the authors have shown the improved robustness of their proposed method on numerous problems. ", "questions": "- Could you elaborate more on why Figure 1 (d) is not invariant? \n- Could you provide more details on the proof of Corollary 1? Why conditioning on W would make the paths blocked?\n- What are the advantages or disadvantages of the penalized problem over data augmentation?\n- The MNIST and Fashion-MNIST datasets are good sources to do the comparisons and rather easy examples to demonstrate the distribution and covariant shifts. I was wondering if the authors can provide few more sophisticated examples to demonstrate this problem? Considering the limited time of the authors I would be very satisfied with a brief discussion on additional one or two more datasets. \n", "limitations": "The authors successfully discussed about some of the limitations of the method in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides an original approach to improve the robustness of anomaly detection models to distribution and covariant shifts on the data. The mutual information based penalty on the latent space is a very creative idea. The authors did a really good job on explaining the problem and motivating their work especially at Section 3.2. Moreover, the analysis of drawbacks of current AD methods such as STFM, Reverse Distillation etc, leaves no questions marks on the purpose of this work and helps readers to easily navigate through the paper. I also found Figure 1 very explanatory (even though I have small question). ", "weaknesses": "As I have discussed in the strength part, I mainly liked the organization of the paper and the authors did a good job on motivating their work. I had hard time to understand the purpose of Figure 2. I would also suggest authors to provide more details on the proof of Corollay 1 and include more explanation on the notations. For the sake of the flow, I would also suggest authors to include the definition of the technical terms such as d-separation, confounding, together wit the notations.  \n\nLastly, I am slightly concerned about the limitations of this method in practice. I am not too much familiar with other AD problems that suits the problem setup given in the section 3.1.; therefore, I could not think of problems other than MNIST and FMINST that this method can be implemented to. I would be very happy if authors can comment on that. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation, constructive feedback, and thorough review of our work. We will address the pointed weaknesses jointly with raised questions.\n\n**I had hard time to understand the purpose of Figure 2.**\nThe goal of Figure 2, was to help visualise the formulation given in Section 3, but also provide additional insights on the causal structure of the AD problem that is essential for the derivation of Corollary 1. \n\n**Changes:** We have expanded Figure 2 with Z now being presented in the graphic, and will refer to it in Corollary 1 for its independence assumption.\n\n**On the invariance of Figure 1 (d).**\n We thank the reviewer for overall positive feedback on Figure 1. \nRegarding the posed question, through our formulation, in the context of AD, we can only guarantee invariance is attained within the support of p* (W=0). Invariant representation of each p_do(E=e_i) with respect to the environment variable, would mean that normal samples (depicted as the circles) are being mapped to the same region of $Z$ irrespectively of the intervention to the variable E. In (d), an contrarily to (b), we observe that the distributions green, blue, and red, do not overlap inside the support of p* (in yellow). \n\nFor an empirical example, we have added a TSNE plot of the embeddings of a deep anomaly detector under different levels of regularization (see global response). \n\n**Could you provide more details on the proof of Corollary 1?**\nWe have added a more detailed proof of Corollary 1. In particular, we explain why all paths from Z to X_e are blocked. We give in appendix a recap on d-separation and confounding. We make it clearer that when f learns invariant representations, f must be X_a-measurable, and therefore, X_e cannot influence Z, which removes the arrow from X_e to Z in the diagrams. This shows then that any path from Z to X_e is blocked. \n\n**Why conditioning on W would make the paths blocked?**\nConditioning on W makes W observed. Note that any path that goes from Z to E goes either through W or through X_e. Paths through X_e are blocked, because X_e does not influence Z, as Z = f(X) is X_a measurable. Paths through W are blocked, by d-separation, because W is observed and in none of those paths, the arrows are all pointing to W.\n\n**What are the advantages or disadvantages of the penalized problem over data augmentation?**\nThis has is an interesting question that we have also posed ourselves.\n\nAlthough data augmentation is a well-established technique to rely on in the context of model robustness to distribution shifts, there are two main problems with solely relying on data augmentation to solve a problem of distribution shift. First, data augmentation relies on both knowing of the existence of a distribution shift, and a way to simulate it. This could be achievable when the distribution shift is characterized by transformations in easily identifiable attributes (e.g. the color of a digit), and which are also easy to simulate through data transformations. However, in real-world settings, this is rarely the case: distribution shifts are complex transformations that are almost impossible to identify, and in many cases impossible to programmatically simulate. For example, in the case of the Camelyon17 dataset that considers histological cuts from different hospitals, the changes between environments that are derived from the change in histological staining are easy to visually inspect, and could even be simulated. But, by changing the hospital, the patient population also changes, and with it several cofounders, such as group age or patient comorbidities, that are both impossible to know without further annotation and unfeasible to simulate. Furthermore, data augmentation does not induce an invariance to a particular distribution shift, giving no additional “guarantees”. It only increases the pool of examples of the data in hopes that the model implicitly captures the additional simulated variance in the images. \n\nYet, data augmentation could still be beneficial in the context of penalized invariant regularization, by producing meaningful augmentations in the context of a multi-environment setting. It could be used to generate additional data for a single intervention (increasing the pool of available samples of a specific environment), or be used to generate new data for a new intervention (increasing the number of environments available, and the overall pool of samples in the dataset). This would effectively alleviate the second drawback of solely using data augmentations. \n\n**I was wondering if the authors can provide few more sophisticated examples to demonstrate this problem?**\nWe appreciate the reviewer's request for a more realistic evaluation setting.\nFirst, we would like to highlight that we already include in our work experiments in Camelyon17. This is a real-world dataset of medical images where we tackle AD for predicting the existence of tumorous lesions in histological cuts, whilst subject to a domain shift derived from a change of hospital. Although, as mentioned in the previous point, the precise nature of the features changed when intervening in hospital variable, by employing our partial conditional distribution regularization we are able to induce some invariance to the distribution shifts, as demonstrated through our improvements of 2 to 8% in the o.o.d AUROC. \n\nAlso, we have included another real-world natural image dataset: the Waterbirds dataset. This is a realistic dataset where the distribution shift is induced through changes in the habitat depicted in the image background. Similarly to Camelyon17, we also demonstrate an improvement in the robustness of the AD models when accompanied by our proposed regularization term. For more details on this experiment, we refer the reviewer to the global comment.  \n\n\n\n\n\n"}, {"review_id": "xxYPT1uZIe", "reviewer": "Reviewer_AUeF", "summary": "This paper tackles the problem of anomaly detection under a distribution shift. The authors describe the setting in detail, where the training is drawn from multiple subsets of examples, and test data is drawn from different subsets. The subset of each training example is known. The goal is to detect anomalies which are not subset specific. The authors then present a regularization term that aids finding anomalies which generalize over different subsets.", "questions": " ", "limitations": "1.\tThe setting requires a labelling of the domains / nuisance attribute during training.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1.\tThe tackled problem is important for real-life applications (as presented by the authors).\n2.\tThe presented method is simple to implements and shows a notable and consistent improvement on top of all other methods.\n", "weaknesses": "Overall I find the idea interesting, yet I have 3 main concerns regarding the paper: (i) Writing quality. (ii) Setting and mathematical formulation novelty. (iii) Benchmarked datasets.\n\n1)\tSetting and Mathematical Formulation: In the paper the authors claim to present a novel setting for anomaly detection as a key contribution (contribution #2, lines 70-71). Yet in fact, a similar setting was presented in Red PANDA when treating a nuisance feature as a sub-domain. Moreover, the mathematical formulation is almost identical to the one presented in DCoDR [], which Red PANDA is built over, but no attribution is mentioned. I believe claiming novelty on the presented setting is unjustified. Also, using the formulation used in Red PANDA can save some of notations presented, which complicated the writing.\n2)\tWriting Quality: I believe the paper is not written well, most noticeable the experimental section (Section 5). It is hard to follow to exact experimental design of the authors, and some key parts are missing in its description: (i) The experimental design of the synthetic benchmarks is unclear about how many subsets appear in the training data (important for baselines such as Red PANDA). (ii) The results are presented in bar charts, making it impossible to get an exact evaluation. (iii) Each method is evaluated by 4 different bar chart results. I didn’t see any part in the paper (or the supplementary materials) that clearly explains what each result describes. (iv) The proposed method is an addition of a regularization term to other existing AD methods. While the term itself is described clearly, the adaptation of other AD methods is not.\n3)\tDatasets Selection: Another concern regards the dataset used for the evaluation in the paper. I believe the method should be tested in more challenging scenarios, as presented in Red PANDA. The Edges2Shoes and SmallNORB datasets should be much more challenging than MNIST and Fashion-MNIST.\n\nI have also some other minor concerns:\n1)\tIn the ablation study of the regularization weight (lines 273-282) states “This pattern suggests a simple linear search could be sufficient to identify an optimal weight for the regularization term”. I believe this statement is ill-posed, as this is only true if given some labelled anomalies for constructing the test set. In real-life scenarios, this is not always the case. \n2)\tMinor Mistakes: \na.\tLines 131-132: all definition refer to f_1.\nb.\tI believe the notation of E in line 192 should be X_e.\n\n\n[1] Kahana, Jonathan, and Yedid Hoshen. \"A contrastive objective for learning disentangled representations.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thorough review of our work, as well as the comments and proposed suggestions. \n\n**1. (i) Yet in fact, a similar setting was presented in Red PANDA when treating a nuisance feature as a sub-domain.**\nWe agree with the reviewer that nuisance features when accompanied by a particular intervention to a covariate is part of our setting. In particular, a nuisance feature can be treated as a shortcut feature, i.e. a stylistic feature that could help the model to perform well in-distribution but fails in out-of-distribution generalization. We provide further analysis of shortcut features and of their vulnerability to covariate shifts in Section 3 of the Appendix. \n\n**Changes:** We will update our manuscript by describing the connection of how our work tackles shortcut features, to the analysis of nuisance features as described by Red Panda.  \n\n**1. (ii) Moreover, the mathematical formulation is almost identical to the one presented in DCoDR [], which Red PANDA is built over, but no attribution is mentioned.**\n\nWe thank the reviewer for bringing to our attention the connection between our work and the work in disentangled representation learning, DCoDR.\n\nAlthough we acknowledge that DCoDR poses a double requirement of invariant and informative representations, its mathematical formulation of invariance is different from ours and entails an alternative take on invariant representation learning. In particular, it defines the invariance condition as $p(E|Z) = p(E)$, where $E$ denotes the environment and $Z$ is the learned representation. It then imposes a separate condition *per domain/environment*, such that $P(Z|E)=U_s$, where $U_s$ is a uniform spherical distribution, which is enforced through a contrastive loss term. On the other end, using the *do* notation, we formulate our invariance condition through a pairwise equality across environments, as $p^{do(E=e)}(Z)=p^{do(E=e')}(Z)$. This formulation ultimately leads to our partial conditional distribution invariance, a distinct condition from the one formulated in DCoDR.\n\nFurthermore, we highlight that contribution #2 covers robustness to distribution shifts, both covariate and domain shifts, and more importantly, the mathematical formulation of an anomaly detection problem under causal inference - which neither DCoDR nor Red PANDA do. \n\nTherefore, we argue that contribution #2, covering both setting and mathematical formulation of distribution shifts through the lens of causal inference, is a novel contribution to the field of AD. \n\n**Changes:** We will reference DCoDR as part of our related work (Section 2.2.) as an example of a recently proposed and effective invariant representation learning method. \n\n**2. (i) The experimental design of the synthetic benchmarks is unclear about how many subsets appear in the training data.**\nAlthough the number of environments included in the training data (two) of the synthetic covariate shift experiments can be found in line 23 of the Appendix (as well as further details on the data generation process), we thank the reviewer for pointing out that this detail should also be in the main manuscript. \n\n**Changes:** We have updated lines 236-237 as follows:\n*\"The training data is comprised of two distinct environments, generated by manipulating original instantiations of each factor, and ensuring all factors differed between the two environments. For the test data, we defined five distinct environments denoted as $e_0, e_1, ..., e_4$. The environment $e_0$ corresponds to images from MNIST for a specific digit under an original instantiation of each factor from one of the two training environments.\"*\n\n**2. (ii) The results are presented in bar charts, making it impossible to get an exact evaluation.**\nWe agree with the reviewer on the need to provide exact values for all experimental results. We chose to present the results in bar charts for better interpretation and comparison between methods.\n\n**Changes:** We will add all the results in table format to the Appendix. \n\n**2. (iii) Each method is evaluated by 4 different bar chart results. I didn’t see any part in the paper (or the supplementary materials) that clearly explains what each result describes**\nTo better visualize the drop in performance from in-distribution to out-of-distribution, we have overlaid the bar plots. In all figures the in-distribution setting corresponds to the transparent bar plot, and the out-of-distribution corresponds to the fully opaque bar plot. This is information was provided only in the caption of Figure 3.\n\n**Changes:** We will update the captions of Figures 3 and 4 with this additional information (see caption of results for the waterbirds experiment), and add these details to the text of Section 5.2.\n\n**2. (iv) The proposed method is an addition of a regularization term to other existing AD methods. While the term itself is described clearly, the adaptation of other AD methods is not**\n\nAs covering the details of each adapted method would span too much content in the main manuscript, we have provided all implementation details for all methods in Section 5 of the Appendix, including hyperparameters, anomaly scoring, and backbone choice.\n\n**3. I believe the method should be tested in more challenging scenarios (...).The Edges2Shoes and SmallNORB datasets should be much more challenging than MNIST and F-MNIST.**\n\nWe agree with the reviewer's assessment of MNIST and F-MNIST. We would however point out that, besides these two datasets, we have also evaluated all methods in Camelyon17. This is a challenging and widely used benchmark for domain generalization, in a real-world medical imaging setting. \nMoreover, we have added now experiments on an additional dataset with real-world natural images: the Waterbirds dataset (see global comment). In these, we also demonstrate the shortcomings of all baseline methods (including Red PANDA) and the benefits of introducing our regularization term. "}]

Author rebuttal:
We thank all reviewers for their time and constructive feedback when reviewing this work. 

In this global response, we will provide details on the additional experiments performed and address the general concern of partial conditioning on only W=0.
 
**1. Experiments on Waterbird**
To address the request of the reviewers for an additional experimental setting, we have also evaluated our method in the Waterbird dataset [A]. This is a real-world natural image dataset where the distribution shift occurs as the natural habit depicted in the background changes between an aquatic and land setting. From the two kinds of birds in the dataset (water and land birds), water birds were assigned to training data and land birds were set as an anomaly. To make this a more challenging setup, we have defined a highly unbalanced distribution of the environments, with 184 images in training data with a water background and 3498 with a land background. The evaluation of the methods was performed in images with only a water background for out-of-distribution and only land background for in distribution.

Looking at the Fig. 1 in the attached .pdf we observe that in all original unregularized methods evaluated, there is a drop in performance when comparing in-distribution (background transparent plots) and out-of-distribution (foreground opaque plots). The performance gained by adding partial conditional invariant regularization to each method ranges from 5% to 15% AUROC. 

**Changes:** These experiments will be added to the main manuscript.

[A] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." ICLR 2020

**2. Experiments with pre-trained invariant encoders**
We also extend our experimental evaluation by adding a comparison to [B], an environment-aware framework for AD that pretrains the encoder of the AD model using an invariance-inducing method (LISA or IRM). We evaluate this method in MNIST and F-MNIST subject to targeted covariate shifts (the exact same setup as seen in our original experiments). The method was incorporated into all baselines (SOTA AD methods) and compared to the same baselines while regularized through partial conditional invariance. 

We show the results of these experiments in Fig. 2 of the attached .pdf. 
Across all methods tested, and in both datasets, we observe a sharp decrease in performance when compared to a baseline non-invariant method and that it provides less robustness to covariate shifts than our proposed methodology.

[B] Smeu et al. "Env-aware anomaly detection: Ignore style changes, stay true to content!." arXiv (2022)

**Changes:** These experiments will be added to the main manuscript, and the framework described in the Related Work section. 

**3. Additional details on partial conditioning (W=0)**
It is unclear if partial conditioning on W=0 is enough for all settings of distribution shifts AD. We agree that partial conditioning on W=1 would be better, however, this is not possible as partial conditioning on W=1 requires a representative sample of anomalies. 

Note, however, that even partial conditioning on both W = 0 and W = 1 would not be enough to attain invariant representations, from a theoretical point of view. The state of the art has only been able to provide necessary conditions for learning invariant representations. This is reflected by Corollary 1, where we only offer necessary conditions for invariant representations. This corollary derives from the recent works from (Veitch et al). Sufficient conditions for learning invariant representations are still unknown and remain a challenge for future work.

Still, we want to provide intuition for cases where partial conditioning is enough. We believe that the main condition to understand the limitation of the invariance of (W=0) is the behaviour of the style feature across the two instantiations of W. In particular, how disentangled these two variables are in terms of their information content. For example, in MNIST, in a setting where the colour of the background changes, both normal images and abnormal images will have the information concerning their digit remain the same. Therefore, in this setting, if these two variables are captured well enough by the un-regularised AD method, attaining invariance to the background colour with W=0 could also lead to robust behaviour for W=1. To some extent, we also hypothesize that the limitation of the partial conditional invariance is dependent on how the final representation achieved by the encoder is able to disentangle the factors from $X_e$ and $X_a$, and could see improvements as general representing learning advances.

Furthermore, we believe there is an inherent benefit to being agnostic to the outlier process (W=1) when inducing an invariance condition in AD, as any assumptions concerning the characterization of an anomaly are not falsifiable during the training process.

Finally, one of our contributions is to demonstrate empirically that learning representations with just partial conditioning on W=0 is still an effective way of attaining invariant representations. Camelyon17 is a dataset where the mechanisms connecting the environment variable and the abnormal class are less clear, in particular, how the physiological characteristics of the population in the out-of-distribution environment affect the tumorous case. However, in this setting, we are still able to considerably improve the robustness of all evaluated models by only relying on partial conditioning on W=0. 

**Changes:** We will add a sub-section to the discussion with the limitations of partial conditioning (W=0). 

**Final remarks**
We believe to have addressed all questions and weaknesses raised during the review process. We would greatly appreciate it if the reviewers responded to the proposed change, and would kindly ask you to adjust your review score while taking the rebuttal into account.



Meta-review: The paper addresses the problem of anomaly detection under distribution shift. Three of the reviewers (zVDG, yiE8, ZicU) have chosen to accept the paper, with various degrees of enthusiasm. The reviewers appreciated the way in which the problem was formalized and motivated, the use of the MMD regularization, with one of the reviewers seeing the approach as creative. The method was also noted by a different reviewer as "pioneer in studying OOD detection from the causal perspective", an assessment which I have come to agree with. Two of the reviewers (ePdN, AUeF) have chosen to borderline reject, with the discussion not changing their opinion. On the other hand, the reviewers questioned whether other existing AD methods could be better adapted for the same setting. A concern that was brought up was the similarity to Red PANDA.The authors made a convincing case, in the discussion, that the theoretical justification for the partial conditional invariance is nontrivial, and that the method goes beyond simply handling spurious correlations and also outlined the differences between their approach and Red PANDA/DCoDR, which I found to be compelling. This is a new method, addressing a very specific problem that was previously not fully considered, let alone solved. 

Finally, as the reviewers have not reached consensus following the discussion period, I read the paper myself. I found that it clearly illustrates the problem, makes a compelling case why invariance should be a factor in training anomaly detection models, presents a pertinent method to solve the problem and demonstrates its use on a number of datasets. As such, it should be of interest and of potential use to a segment of the NeurIPS community.
Decision: Accept (poster)