Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data
Authors: Robert Schmier
Ullrich Koethe
Christoph-Nikolas Straehle
Venue: Accepted by TMLR ()

Abstract: Detecting test data deviating from training data is a central problem for safe and robust machine learning. Likelihoods learned by a generative model, e.g., a normalizing flow via standard log-likelihood training, perform poorly as an outlier score. We propose to use an unlabelled auxiliary dataset and a probabilistic outlier score for outlier detection. We use a self-supervised feature extractor trained on the auxiliary dataset and train a normalizing flow on the extracted features by maximizing the likelihood on in-distribution data and minimizing the likelihood on the contrastive dataset. We show that this is equivalent to learning the normalized positive difference between the in-distribution and the contrastive feature density. We conduct experiments on benchmark datasets and compare to the likelihood, the likelihood ratio and state-of-the-art anomaly detection methods.

Published in Transactions on Machine Learning Research (04/2023)
Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data
Robert Schmier
robert. Shbert.cchmmer&@ddeboocc.
Bosch Center for Artificial Intelligence, Renningen, Germany University of Heidelbery, Germany
Ullrich Koethe
University of Heidelberg, Cermany
Christoph-Nikolas Straehle
chrritophh-nikolla.stteeekee
Bosch Center for Artificial Intelligence, Renningen, Germany
Reviewed OpenReview: https: // openreview. net/ formm? id= B4J440NNNj4
httpe://openrevieeu
Abstract
Detecting test data deviating from training data is central problem for safe and robust machine learning. Likelihoods learned by generative model, e.g., normalizing flow via standard log- ligllikli training, perform poorly as an outlier score. We propose to use an unlabelled auxiliary dataset and a probabilistic outlier score for outlier detection, We use a self-supervised feature extractor trained on the auxiliary dataset and train a normalizing flow on the extracted features by maximizing the likelihood on in-distribution data and min- imizing the likelihood on the contrastive dataset. We show that this equivalent to learning the normalized positive difference between the in-distribution and the contrastive feature density. We conduct experiments benchmark datasets and compare to the likelihood, the likelihood ratio and otate-of-the-art anomaly detection methods.
1 Introduction
If an image at test time is not similar to the training images, but stems from another distribution, classical neural networks may classify the image incorrectly with high confidenco(Heaven et al. 2019 Boult et al. 2019). The delection of potential anomalies, also called out-of-disttibuttion delection or outlier delection, is central problem in modern machine learning and is crucial for safe and trustworthy neural networks: At test time, want to know if given input stems from the same distribution the training data, in order to know if we trust our trained net that input.
This may not significantly influence the prediction performance at test time, either because anomalous events are that rare, or because the test data is similar to the training data, but introduces a major security risk: If the camera of a self-driving car has a malfunction or something is occluding the view, the car should be able to detect the situation as anomalous and should not use the input of the camera. The use of outlier detection to improve performance for rare events or unseen data is manifold: One can use outlier detection while training to purify the data set (Zhao et al., 2019b), at train time to give rare training data points stronger training signal (Steininger et al. 2021). or at test time to find and react to anomalies (Wang et al. 2021) In contrast to discriminative networks, where the texture [Geirhos et al. 2020) or the background (Beery et al. 2018) can be sufficient to get satisfactory performance, for outlier detection the network must distinguish between the in-distribution data and all other -unknown- data. The network therefore must 'understand" what the in-distribution data charactorizes and following Richard Feynman's famous saying 'What I cannot create, I do understand', we believe that generative models are therefore the most promising approach to outlier detection. We turn to the fast growing field of normalizing flows (Kobyzev et. al. 2020 which

---

Published in Transactions on Machine Learning Research (04/2023)
Standard Normalizing Flow
Likelhood Ratio
Contrastive Normalizing Flow
Non-animal casses
1.2
Other Animals
1.0
Difficult OGD (horse)
1.25
0.6
Distribution (deer)
0.8
3.00
0.6
0.75
0.4
0.50
0.2
outlier score plk
outlier score -lagpp00/q(x))
outlier score -log (0g - log. (olx) q(0))
Figure 1: Histogram of the outlier scores of all CIFAR-10 classes under models trained with the inlier class deer. The semantically similar classes deer and horse are difficult separate for all methods, our method 'contrastive normalizing flow' (right) achieves good separation for the other classes. We group the other UI classes for this figure into animal (frog, cat, dog and bird) and non-animal super-class (truck, car, plane and ship) for better readability. We abbreviate log+ (-) as the logarithm of the positive difference for the outlier score of the contrastive normalizing flow.
allow exact density estimation, fast sampling, and suffer (almost) no mode collapse. While an important line of rescarch deals with detecting small anomalies within an image, so-called defects, in industrial scttings (e.g.. Roth et al. 2022) ) we focus in this work on semantic single-shot outlier detection:
Task The goal is to find outliers on image level: An outlier is an image of a different semantic class, which is not known at training time. We train e.g., solely on images of the class deer of the CIFAR-10 dataset and want to distinguish between images of the in-distribution class deer and the other CIFAR-10 classes at test time. In contrast distribution-based outlier detection where set of many test-samples, measurements is compared to the training distribution (see e.g. Coluccia et al. (0133); Bouyeddou et al. (2018)) we focus the single-shot setting, where the inlier vs. outlier decision must be made independently for individual data points. Finding semantic outlier images is a common task definition in the outlier detection literature, e.g..
Ren et al. ) Schirrmeister et al. (0020) Nalisnick et al. 2019b a:: Kirichenko et al. 2020 Choi et al.
2018).
Approach Inspired by the work of Ren et al. (2019) and Schirrmeister et al. (2020), who used the ratio of likelihoods for outlier detection, we modify the training objective for normalizing flows to learn the positive difference of two distributions: An inlier distribution and an contrastive distribution We prove that this is achioved by maximizing the log- liceliioddood on data which stems from p while minimizing the log-likelihood on data stemming from q. We use the negalive log-likelihood under this usually intractable difference distribution given by our normalizing flow as outlier score and demonstrate that this new score especially suited for outlier detection. We argue and demonstrate experimentally that the difference p q is much more robust than the ratio p/q: In regions where p and q are small, there are only few training instances, and the estimators of and q will accordingly be very noisy. The quotient between two noisy small numbers can become very big, whereas their difference remains small.
Moreover, we run experiments comparing different contrastive datasets and show that our method is robust under the choice of the contrastive distribution. We show that our method works well with an contrastive distribution broader than the inlier distribution, and it does not harm the performance if the contrastive distribution contains inlier data. We do not use label information, but see the contrastive dataset a col- lection of unlabeled images. We motivate the use of such auxiliary dataset twofold: Collecting unlabeled images is casy and cheap, but labeling them would be expensive. Since applying normalizing flows on images is computationally intensive, we do not Irain directly on images but employ MoCo He et al. (2020), dimen- sionality reducing feature extractor trained in a self-supervised manner on IMAGENET (Deng et al., 2009) This fits the ongoing development of employing pretrained models to exploit prior knowlodge (Bergman et al. 2020). A schematic overview of our method is shown in figure To give an introductory example, we compare the outlier scores for all CIFAR-10 classes given by standard normalizing flow, the ratio method
2

---

Published in Transactions on Machine Learning Research (04/2023)
and our proposed method trained the deer class of the CIFAR-10 dataset as in-distribution data and IMAGENET contrastive dataset in figure Our method separates semantically distant classes better than existing methods.
We focus on normalizing flows as density estimators, since they are universal approximators with stable training procedure. However, the new objective and theory applicable any density estimator trainable with explicit maximum likelihood, as required for our new objective in equation 3 This includes Mixture- of- Gaussian density estimators binning methods in lower dimensional settings.
Our method 'Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data' combines a generative model with exact density estimation (normalizing flow) and new training ive for density estimators. We employ a pre-trained feature extractor for further performance improvement. Our contributions are:
A new likelihood-basel outlier score
A new objective to train density estimators
Theoretical results relating our new objective to negative log-likelihood training of an intractable difference distribution
Experimental validation of the theoretical advantages compared to the Aikelihood-ratie method
State-of-the-arl results in experiments on standard semantic outlier detection benchmarks
2
Related work
Outlier detection, also known as ont-of-distribution or anomaly detection, is the task of detecting unknown images at test time. See Salehi et al. (2021) for a extensive discussion of the field. In this work, we will focus on semantic outlier detection, by either using one class of given dataset inliers and all other classes anomalies (one-vs-rest setting) or by using complete dataset inliers and comparing against other datasets (dataset-ve-datasst setting).
Outlier detection
Most related work can be categorized as either reconstruction based or using a representation ansatz: In a reconstruction approach, the model tries to reconstruct a given image, and the score relies on the difference between the reconstruction and the original image. This idea goes back to Japkowicz et al. (1995) and has been applied to various domains, e.g. lime series (Zhang el al.] 2019), medical diagnosis {Lu and Xul 2018)) and flight data (Memarzadeh et al.) (2020). Recent models used for image data are reconstruction with a memory module in the latent space (Gong et al.) 2019) or combinations of VAE and GAN (Perera et al. 2019h. An interesting new idea is the combination of VAE and energy-based models by Yoon et al.) [2021] where the reconstruction loss is interpreted as the energy of the model. For the representation approach the model tries to learn a feature space representation and introduces a measure for the outlierness in this representation: Ruff et al. (2018) map all data inside a hypersphere and define the score as the distance to the center. Another approach uses transformations either define negatives for contrastive learning (Tack et al., 2020) or directly train classificr (Bergman and Hoshen 2020). Zong et al. (2018) fit a gaussian mixture model to the latent space of an autoencoder.
Normalizing flows While most generative models are either able to easily generate new samples (like GANs VAEs) or give exact (possible unnormalized) density estimation (like energy-based models), Normalizing flows, first introduced by Rezende and Mohamed (2015) are able to perform both tasks. Nor- malizing flows rely on the change of variables formula to compute the likelihood of the the data and therefore need a tractable Jacobian delerminant: Most normalizing flows achieve this by either using mathematical "tricks" (e.g., Rezende and Mohamed (2015)) ), as autoregressive models which restrict the Jacobian to a tri- angular shape (c.g., Kingma et al. (2016) or by special architectures which allow invertibility (e.g., RealNVP by Dinh al. (2017)). RealNVP makes use of coupling blocks apply an affine transformation on subset of the features conditioned on the subset's complement.
3

---

Published in Transactions on Machine Learning Research (04/2023)
NEGATIVE LOGLIKELIHOOD (IN-DIST)
IN-DIST DATASET
FEATURE EXTRACTOR
NORMALIZING FLOW
CONTRASTIVE DATASET
NEGATIVE LOGLIKELIHOOD (CONTRASTIVE)
Figure 2:
Schematic overview of our method. Standard normalizing flow in black. The feature extractor is an optional extension, which further improves the performance for image data. We do not use an feature extractor for tabular data and for our toy experiments.
Density of normalizing flows for outlier detection The use of generative models seems like perfect fit. for outlier detection: While Classifiers tend to focus on features to distinguish the given classes, generative models need to include all relevant information to be able to generate new samples. Methods with exact density estimation (e.g., normalizing flows, energy-based methods) directly offer a good outlier score via the learned density. Unfortunately, normalizing flows work poorly in the case of outlier detection when employed directly on images: Various works showed that the likelihood is dominated by low-level statistics (Nalisnick et al. 2019a] and pixel-correlations (Kirichenko et al. 2020 and fail to detect anomalies (see also Zhang et al. [2021] (). There have been multiple attempts to improve the outlier score directly by introducing a complexity
measure (Serrà et al. 2020). using typicality (Nalisnick et al. 2019b using hierarchies (Schirrmeister et al.
2020 or employing ensembles (Choi et al. 2018).
Likelihood ratio for outlier detection Another line of research is using the ratio of two likelihood models p/g as outlier score: Ren et al. (2019) use an augmented version of the in-distribution as con- trastive distribution and computed the likelihood ratio with Pixel-CNN, an autoregrossive generative model. Schirrmeister et al. (2020) Irain separate normalizing flows on the in-distribution data and Tiny Images contrastive dataset. They require the in-distribution dataset to be included in the contrastive dataset to meet their hicrarchical principle.
Feature extractor With the broad availability of trained deep models and their good generalization capacity, the use of such models as feature extractor has gained popularity: Most implementations use the auxiliary datasct (e.g. IMAGENET
output of intermediate layers of a model trained discriminator
by Deng et al. (2009)) features: Prominent examples are ResNet (He et al. 2016 used by e.g., Cohen and Hoshen (2020) and Roth et al. (2022), or ViT (Dosovitskiy et al.) 2021 used by Cohen and Avidan (2022) for outlier detection. We use a feature extractor trained in a self-supervised fashion via a contrastive learning objective (Chen et al., 2020). Their and the improved method by He et al. (2020) showed remarkable results
on down-stream tasks without relying on label information.
Combining feature extractors and normalizing flows An existing line of work uses the representa- tions given by pretrained feature extractors to detect outliers: Cohen and Hoshen (020)) use the intermediate layer of a pretrained ResNet the feature representation and use the sum of distances to the k-nearest Neigh- bours as their outlier score. Yu et al. (2021) and Rudolph et al. (2021) train unconditioned normalizing flow on a feature space of a pretrained feature extractor trained on IMAGENET, while Gudovskiy et al. (2022) use conditional normalizing flows (Ardizzone et al. 2019). All of these methods focus mainly on defect detection and localization, while our paper works on the task of semantic outlier or anomaly detection. In contrast to MoCo (He et al. (2020), which is used in this work for the feature extraction, all their feature extractors are trained in a supervised fashion.
4

---

Published in Transactions on Machine Learning Research (04/2023)
3 Background
3.1 Normalizing flows
Normalizing flows are a class of generative models, which allow exact density cstimation and fast sampling. A comprehensive guide to normalizing flows can be found in Papamakarios et al. (2021b). A normalizing flow maps the given data via an invertible transformation T to chosen base distribution of the same dimensionality, e-g-; multivariate normal distribution by minimizing the empirical KI--Divereence between the transformed data distribution in the latent space and the base distribution. This is equivalent to minimizing the negative log-likelihood of the training data under the model. This likelihood can be calculated by the change of variable formula. Therefore, the Jacobian of the transformation needs to be tractable. For the normal distribution as basc distribution, the likelihood can be computed:
= 117o [o TEX 2
L( = Σ log (x)) xEX
-
log JJac,, [a)),(x)) + C,
(1)
where X is the training data, O are the parameters of the invertible transformation T, Jacr denotes the determinant of the Jacobian and C' a constant. independent of 0. To apply the change of variable formula in equation [ the transformation has to be invertible and needs a tractable determinant of the Jacobian. We achieve this by using the realNVP architecture established by Dinh et al. (2017
3.2 Application of feature extractor and MoCo
Standard normalizing flows, c.g.: Ardizzone al. 2019); Dinh et al. (0177; Kingma and Dhariwal (2018) are directly applied on the data space to allow exacl densily estimating and sampling of new data following the in-distribution. As we are interested in outlier detection and do not need new samples generated by the model, we follow a different approach: We employ our density estimation on a pretrained feature space. The outlier detection capability of a normalizing flow is improved in this case, as demonstrated by Kirichenko et al. 2020p. We verify this by running experiments without the use of a feature extractor in the appendix for standard outlier methods (section A.3 and flow methods (section A.9
Additionally, using feature extractor leads to lower computational costs: While for GLOW normalizing flow (Kingma and Dhariwal 2018 D a simple forward task with a single IMAGENET sample takes 162 ms + 1.91 ms, our method (a frozen MoCo feature extractor and a Freia (Ardizzone et al. 2018-2023) AllnnOne normalizing flow in 128 dimensions) lakes only 11.4 ms + 0.25 ms. This difference increases for a training loop (ours 15.5 ms + 0.721 ms versus GLOW 338 ms + 2.45 ms).
We use a pretrained MoCo (He et al., 2020) feature extractor, which has been trained with a self-supervised objective on IMAGENET. Self-supervised rastrastive methods learn representation by maximizing the similarity of different views of the same image in the feature space. For a training step, every training image is augmented twice, by augmentations consisting of color distortions, horizontal flipping, and random cropping. These augmented images are fed into an encoder network and for a positive pair zo. z1 and negative representations 22,..., ..., NN (other augmented images) the infoNCE loss is optimized for all augmented images:
exp(sim(Zo. 21)/T)
t llg .
(2)
Ex exp(sim 2;)/T)
is a temperature scalar and sim(x, y) denotes the cosine similarity (cosine of the angle) between and y. Oord et al. (2018) proved that optimizing this infoNCE loss is maximizing a lower bound to the mutual information between the input data and the learned representation. As we are intorested in somantic anomalies of the data distributiinn, this maximal informative representation is well suited for our method. While MoCo trained without any label information, one could also use e.g., a classifier trained on labeled auxiliary dataset feature extractor. To show the advantage of the MoCo feature extrator over a supervised feature extractor trained without the infoNCE, we an ablation experiment on inceptionV3 and report the results in the appendix in A.4
We the GLOW pytorch implementation of http:///ttta...ccrrrraaallaagggp and used the default hy- perparameters.
5

---

Published in Transactions on Machine Learning Research (04/2023)
4 Method: Contrastive normalizing flow
We introduce our method 'contrastive normalizing flow' as novel way to train normalizing flows: We adapt the training of normalizing flows by maximizing the log-likelihood on in-distribution data æ ~ p and minimizing the log-likelhhood on a broader contrastive dataset y q. The intuition behind this approach is simple: We want the model to learn high likelihoods on in-distribution data and low likelihoods everywhere else. We use the negative log likelihood given by the model trained with this novel training objective outlier score. We train our model with the following objective:
= min Eg po(p))) - Ey log po (3)].
(3)
Rearranging the loss shows that instead learning p, we learn a truncated version of p - to the region where p > q:
L min- - /
da
(4)
logro())(p(x)) u(x)]dd + mjn / logpa(z)lu(2)) p(a))de
min-
(5)
supp(p>a)
supplo><)
min E Cl>C(pp-) log po + min 0
logpo(>))a(a)) - p(ar))aæ,
(6)
0
supp(?><)
In the second term, 9 > p. In this case, L is minimized by Pe + O. In words: The learnt probability trics to put no mass to the learned distributions in regions where there is more contrastive data than in-distribution data.
The first term in equation G is the negative log-likelihood objective for the normalized positive difference distribution = C(p 4)
{p>g) with new normalizing constant C. This term is maximized by with pa(=)) C(p q) Vo E supp(p 4) PPapamakarios et al. 2021a).
By training our model with equation 3 we learn the normalized positive difference between in-distribution and contrastive distribution. This is in contrast to a standard normalizing flow, where we learn the distri- bution of the in-distribution samples. We use the negative logarithm of this new otherwise intractable difference density as outlier score. The pseudocode for our training procedure is given in the appendix in section A.1 We clamp log po()) the contrastive dataset during training by threshold in eq I see
appendix 2.1 for a discussion.
When separating the probability density into a semantic part and a low-level pixel correlation part, Ren et al. (2019) argue that the common low-level correlations are the same for all natural images. This strong assumption is not necessary for our theory to hold but can motivate the method and give the reader some intuition. Using this reasoning, we separate the feature dimensions of our data in high level semantic representation s(x) and a low-level feature representation f(x), where the low level pixel feature condit ional distribution p(fls) is the same for all natural images and s) are volume preserving transformation such that p(x) = (z)) s(x)). We can now rewrite the difference between the in-distribution p and the contrastive distribution as
p(ar)
- = = ((f)s)(p(s)
q())).
This results in a low score when either the low-level features have low density given the semantic content of the image or the semantic likelihood of the image is higher for the broader contrastive distribution than for the inlier distribution. We find both cases important classes of anomalies.
Outlier score The outlier score is the negative log density given by our model, which is equivalent to the negative log-likelihood of the positive difference density described equation 0 By setting a threshold 5, all inputs with outlier score greater than 8 are classified as anomalies. For evaluating, we use area under the receiver operating characteristic ( (AUROC).

---

Published in Transactions on Machine Learning Research (04/2023)
Training data
Generated samples
0.6
CONTR
difference
denely
0.2
0.1
Figure Example of an 1D contrastive normalizing flow: We train on samples x ~ p == (o,, as in- distribution and y ~ q = (1, as contrastive datasct. The training distributions p and g are shown on the left. The learned distribution of our contrastive normalizing flow method and the analytic normalised positive difference between the ground truth densities are shown on the right. The black lines connect equivalent points in both diagrams.
Table 1: Comparison of our contrastive normalizing flow with the standard normalizing flow and the ratio method.
Use of contrastive data
Behaviour for p.qæ1
Reduces influence of low-level features
Method
Normalized score
Single model
Standard NF Ratio method
X
stable for pol unstable stable
X
x
Contrastive NF
1D Toy example We conduct a qualitative 1D experiment to demonstrate that our derived theory holds by training a contrastive normalizing flow on samples I ~ p pON((0, 1) as in-distribution and y ~ = (1,2)) as contrastive dataset. We show in figure 3 that the density learned by the model exactly matches the positive difference density described in equation 6 We do an ablation study on the role of the clamping-
hyperparameter in the appendix A.4
Outlier detection and comparison to ratio method We argue that our learned difference distribution is especially suited for outlier detection: When using broader distribution contrastive distribution we can assume that p(x)= > ((x)) holds for in-distribution data. In regions where the contrastive distribution g(x) has high density the learned positive difference will be small or zero. This behaviour is similar to the likelihood ratio method Ren et al. (2019): They train two density estimators, one on the inlier distribution, the other one on the contrastive distribution and use their ratio as outlier score. An advantage of our method compared to the likelihood ratio that our learned density po(x) robust in areas where p(z) and g(x:) are very small, i.e,, where the in-distributiin data, and the contrastive distribution daía are sparse. The learned density pe is normalized and therefore integrates to 1. Thus, in the areas where both distributions and have no support the difference density is zero or almost zero. When comparing - log (a(x)) at test time to threshold to see whether it is an inlier (less than threshold) or outlier (greater than threshold) data points outside of the support of p and will be reliably detected as outliers. In contrast, the density ratio p(=)/q(x)) is ill-defined in regions where both p(x) and g(x) have little or no support. The division of two small numbers can either be very large or very small. This leads to problems demonstrated by the experiments in section 5 because the data which
might want to detect as outliers the contrastive distribution which was used during training the likelihood method fails due to spurious high ratios in the noisy tails of the learned distribution. We show in 2D loy experiment, that the likelihood ratio has the highest inlier score far away from the inlier and contrastive distribution while our model is able to capture the in-distribution well. We show in table [ an overview of the properties our method in comparison to the ratio method and the standard normalizing flow.
be very far from
7

---

Published in Transactions on Machine Learning Research (04/2023)
ground truth samples
learned ratio score
learned contrastive score
105
1.20
contrastive in-distribution
3
1.05
2-
0.90
75
60
0.60
-1
0.45
-2
0.30
-3
15
0.15
-4
0 -4
0.00
-4
-2
2
-2
-2
Figure 4: Learned in-distribution scores for the density ratio method and our contrastive normalizing flow on a 2D toy examples. For better visualisation, we plot the exponential the in-distribution score (equivalent the exponential of the negative outlier score used throughout the paper). On the left the ground truth samples for the in-distribution and the contrastive data shown. While the ratio method has the highest in-distribution scores far away from both distributions in the lower left, the contrastive flow in-distribution score caplures the inliers well.
2D Toy Example To show the theoretical advantage of the learned difference density over the density ratio method, we conduct a 2D toy experiment: We learn contrastive normalizing flow on a normal distribution centered at [1,1] I] as in-distribution and a normal distribution centered on the origin as contrastive distribution. For the ratio method, we train a separate normalizing flow on each distribution, and take the ratio at inference time. As we see figure ! our contrastive uormalizing captures the in-distribution data, while the ratio method has the highest in-distribution scores far from both distributions. This not surprising, as the analytic log ratio of the two Gaussian distributions goes infinity for [-00, oc].
We also show this behaviour in real world setting in section 5.2.2
5 Experiments
Experimental outline First we investigate the performance our new difference method for different choices of the contrastive dataset and compare the likelihood ratios method. We study the case of inliers in the contrastive dataset, the behavior for dissimilar in-distribution and contrastive datasets and the supervised setting, where the contrastive dataset contains some examples of test time outliers. To compare our method to other semantic outlier detection methods, we conduct experiments on CIFAR-10, CIFAR-100 and CelebA.
5.1 Implementation details
To extract useful features, we use MoCo encoder pretrained on IMAGENET (He et al. 2020). We show the inferior performance for another feature extractor in the appendix in section A.4 As features use the output of the network, this is in contrast to the original MoCo implementation, where classification is done via an MLP head on the penultimate layer. Working with the last layer results in better performance for outlier detection. Because the contrastive MoCo objective is invariant under changes of the norm of the features, we normalize all features to a hypersphere and add small noise afterwards to obtain non- degenerated density, we discuss the normalization in the appendix in section [A.4] For the normalizing flow implementation, we use the 'Framework for Easily Invertible Architectures' (Ardizzone et al. 2018- 2023). Unless otherwise specified, we use eight of their "AllinOne" -block for our architecture. We list all hyperparameter for training in section A.2.3 in the appendix. As mentioned in the method section 4 we
8

---

Published in Transactions on Machine Learning Research (04/2023)
clamp the contrastive likelihood to prevent the loss from diverging. We discuss the clamping in the appendix in subsection A.2.1 1 We finetuned the pretrained MoCo feature as discussed in the appendix in A.2.2
5.2 Investigation of the contrastive distribution and comparison to ratio method
To investigate the role of the contrastive distribution and get insights on the difference between our method and the ratio method proposed by Ren et al. 2219 we run ablation studies on different dataset mixtures as rastivvtive distribution. We run most of our ablation experiments on the deer and the horse class, as our model and the baselines struggle to distinguish these two classes (see table 3 for the full confusion matrix), and experiments are most informative here.
5.2.1 Contrastive distribution containing in-distribution data
In this ablation experiment the performance for a contrastive distribution containing in-distribution samples is mcasured. When including inliers in the contrastive datasct, we expect to no change in performance following our theoretical derivation: With in-distribution data and pure contrastive data y (and learned
positive difference p = Cpp 4) p>g)) we get for the mixed contrastive data Jnew = (1 - p)p + pe:
Pnew
C(p Qnew) ) 1 p>> quew
(7)
C(p (1 p)p pe) p0011>>>1111p>p>
(8)
= C(up
(9)
= C'u(p- - 4)1pp99
(10)
= f7
(11)
with Cμ= C. To show this qualitatively, we use the horse class of the CIFAR-10 dataset inlicr class and mixture of IMAGENET and in-distribution data contrastive dalaset. We denote the mixiure fraction of IMAGENET for the contrastive dataset as = pumber of IMAGENET samples so for 1 the contrastive contrastive dataset size datasct consists only of imagenet samples and for = 0 the contrastive dataset contains only in-distribution data. We report the AUROC scores for different mixing fractions js in figure 5 One can see that the performance of our method stays constant for μ > and reliably beats the standard normalizing flow, where the ratio method drops in performance as μ approaches zero and the contrastive distribution contains more and more inliers. For μ 0, as in-distribution and contrastive distribution coincide, both models do not get useful training or detection signal and outlier detection fails for both (AUROC score approx. 50).
5.2.2 Use of a narrow contrastive distribution
In this ablation study, we want to investigate the performance for a contrastive dataset with a small semantic variety, e.g.; a single semantic class. We call such contrastive dataset narrow. We use the deer class of CIFAR-10 as in-distribution data and a mixture of IMAGENET and the horse class of CIFAR-10 as contrastive dataset. We denote the mixture fraction of IMAGENET as ju number of IMAGENET samples dataset size
for /4 1 the contrastive dataset consists only of imagenet samples and for / 0 the contrastive dataset contains only samples from the CIFAR-10 horse class. We plot in figure P the AUROC scores for the horse class included in the mixture and the other CIFAR-10 classes. While the performance of the ratio method drops significantly (AUROC 96.5 to 75.6) when using a single narrow CIFAR-10 class as contrastive dataset (n = 0), our method only drops slightly (AUROC of 96.7 to 93). We plot in figure 7 the histograms for in-distribution, contrastive and out-of-distribution data for only the horse class as contrastive dataset. We see that the ratio method fails and assigns lower outlier score to ont-of-distribbtioo examples than to in-distribution data, especially for the setting of no IMAGENET samples in the contrastive distribution. Our difference method stays almost constant for reasonable mixtures of IMAGENET and the contrastive class. For a small mixture fraction μ,, our model is outperformed by the standard normalizing flow by a small margin because of the overlap of in-distribution and contrastive class in the MoCo feature space. We conclude that our method more robust than the ratio method for outliers far from the inlier and contrastive distribution as discussed in section g and the results of the toy experiment in section transfer to real world data.
9

---

Published in Transactions on Machine Learning Research (04/2023)
OOD performance on easier cases
OOD performance on difficult cases
100
100
ratio
our method
90
90
AUROC score fo hard OOD data
AUROC score OOD data
70
70
60
60
ratio
40
40
our
standard
30
30
0.6 contrastive distr. mixture fraction u of IMAGENET in-distribution data
0.4
contrastive distr. mixture fraction u of IMAGENET vs. in-distribution data
Figure Performance for our model, a standard normalizing flow and the ratio method trained on the CIFAR-10 horse class as in-distribution and of mixture of IMAGENET and in-distribution data con- trastive dataset. lr denotes the fraction of imagenet samples in the contrastive dataset, so for = 1 the contrastive dataset consists only of imagenet samples and for = 0 the contrastive dataset contains only samples from the CIFAR-10 horse class. On the right we plot the AUROC score against difficult out-of- distribution cases (deer class of CIFAR-10), on the left against all other CIFAR-10 classes (without horse and docr). Errors are computed over three different sceds.
5.2.3 Use of test time anomalies at training time
In this ablation study, the performance in a partially supervised setting is examined, where the contrastive data is a mixture of broad data distribution and anomalous data. We use the same setting in the previous experiment: the deer class of CIFAR-10 as inlier and a mixture of IMAGENET and the horse class of CIFAR-10 as contrastive dataset. We measure the outlier performance between the inlier class and the horse class this is in contrast to all other outlier detection experiments in the paper, as we explicitly use splits of the same distribution as contrastive data and training time and anomalous data al test time. We show the performance for different mixture fractions our method, ratio method and normalizing flow comparison in figure 6 We see that the performance of our model increases already with small percentage of anomalous data in the mixture. Both methods saturate at AUROC score of about 95, conclude that both classes have a significant overlap in the MoCo feature space and perfect separation is even in the fully supervised case not possible.
5.2.4 Findings
Using the results of the previous experiments, we formulate the following findings and use them to select a suitable contrastive distribution for the benchmark experiments:
Difference method improves on previous flow methods In all conducted experiments our difference method is par or outperforms the ratio method and improves standard normalizing flows for broad contrastive distributions when the test-time outlier distribution is known at training time. Especially for narrow contrastive distributions, our difference model shows significant improvement. When contaminat- ing the contrastive data with in-distribution data, our method is able to use a small fraction of unknown contrastive datapoints for a stable performance, where the ratio method shows a strong performance degra- dation. Additionally, our model has another interesting advantage: While both methods can use the same architecture, our difference method only needs one model at test time, whereas the ratio method needs separate in-distribution and contrastive models.
10

---

Published in Transactions on Machine Learning Research (04/2023)
Supervised anomaly detection performance
Performance for narrow contrastive dataset
ratio
95
pushdown
95
standard
AUROC score for horse class
AUROC score for ood classes
90
85
85
80
80
75
ratio
pushdown
70
standard
70
contrastive distr. mixture fraction IMAGENET horse-class contrastive distr. mixture fraction y IMAGENET horse-class
Figure Performance for our model, the ratio method and standard normalizing flow trained on the CIFAR-10 deer class in-distribution and mixture of IMAGENET and the horse class of CIFAR-10 contrastive dataset. On the left we plot the AUROC score against the horse class included in the mixture (which correspond to fully supervised setting when μ 0.0), on the right against all other CIFAR-10 classes (without horse and deer). For a contrastive dataset containing only images of the CIFAR-10 horse class (μ = 0.0), the ratio methods fails, when our method only slightly drops in performance. Standard deviations are computed over thrce different seeds. Please note that although both graphs use the same trained model, the experimental setting vastly different: On the left. the distribution of the outliers is already known at training time, where the right stay in the uniformed outlier detection setting.
Do not narrow contrastive distributions for uniformed outlier detection When choosing narrow dataset (e.g,, single other semantic class) contrastive distribution, the performance of our method and the ratio method drops significantly when the contrastive samples differ from the test time outliers (μ = 0.0 in the right plot of figure G). In this setting, both methods are outperformed by a standard normalizing flow trained without the use of a contrastive dataset.
How to choose good contrastive dataset The performance of our method neither improves nor declines when including inlier data in the contrastive dataset. Even for contrastive datasets consisting of ninety percent inlier data, the performance of our difference method stays almost. constant. the uninformed outlier detection setting, a broad contrastive distribution works best, even if it contains some inlier data. Including known outliers However, only using a set of narrow known outliers contrastive data would hamper the performance on other unknown outliers.
the contrastive distribution helps in the informed outlier detection sctting,
Contrastive distribution for benchmark experiments Following our findings, we use IMAGENET as a broad contrastive datasct for our benchmark experiments on CIFAR-10, CIF and CelebA. IM- AGENET is the same dataset used in the training of the MoCo feature extractor.
We use the to disentangle the effect of the fcature extractor and the contrastive normalizing flow and report the results in the appendix in section A.9 We see a similar behaviour for the dependence of the outlier detection performance on the mixture fraction μ with and without the use of a feature extractor, but a performance decrcase in uninformed outlier detection performance without a fcature extractor. The results support our findings in this section and confirm our choice of employing feature extractor before applying the contrastive normalizing flow.
experimental setup but without the use of feature extrator to train directly images
11

---

Published in Transactions on Machine Learning Research (04/2023)
Ratio Method
Contrastive Normalizing Flow
2.00
nndiisribbtton
in-distribution
1.75
contrastive distribution
contrastive distribution
dustr--iistribttion
didittrrbbuinnn
frequency
frequency
0.00
outlier score
outlier score
Figure 7: Histograms of the outlier scores for in-distribution, contrastive and ood data for our model (right) and the ratio method (left) trained the CIFAR-10 deer class as in-distribution and the horse class of CIFAR-10 as contrastive dataset. The out-of-distribution data are all other CIFAR-10 classes. The models coincide with the models for 0.0 in figure 6 While for the contrastive flow in-distribution samples have the lowest outlier scores, the tail of the out-of-distribution data (highlighted with a red circle in the figure) surpasses the in-distribution data for the ratio method and out-of-distribution samples have the lowest outlier scorc.
5.3 Benchmark experiments
We run experiments for two versions of our contrastive normalizing flow: For the contrastive normalizing flow (CF), we train solely with our new objective. We also run experiments using a finetuning training procedure (CF-FT), where we train a standard normalizing flow first and then use our new objective to finetune for two epochs.
Our methods
Baseline methods We used the PyOD library (Zhao et al. 2019a. compare results to standard outlier detection techniques. We present results for PCA, KDE, and KNN conducted on the MoCo feature space, and show additional results for other methods and training directly on the image data (without feature extractor) in the appendix (see A.3h. J. As a simple baseline, we use the mcan squared crror (MSE) to the mean of the feature space representations the training set as an outlier score. This equivalent fitting a normal distribution the feature space around the mean of the training data. This simple baseline already gives good results with the finetuned MoCo feature extractor. We extend the MSE to also employ the contrastive distribution by taking the difference of the MSE to the mean of the inlier set and the rantrastive set as outlier score. This corresponds to the likelihood ratio of two normal distributions. We call this method MSÉ-ratio. As third method (Flow), we train an unconditioned flow with the same architecture as our model on the MoCo feature representation and use the negative log-likelihood under the learned distribution as an outlier measure. As fourth baseline method (Flow-ratio), we train an additional flow (with the same architecture as our model) on the contrastive dataset and use the likelihood ratio of the learned inlier flow and the learned flow on the contrastive data using the hierarchies of distribution method of (Schirmeesser et al. (2220). In contrast to their work, the normalizing flows are trained the MoCo feature space and not the images directly. We also employ the outlier exposure method of Hendrycks ct al. (2019a) on our architecture by training standard normalizing flow on the inlier data and finetune afterwards with their likelihood margin objective using the contrastive dalaset. Finally, we compare our method to "CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances' (Tack et al. 2020). This method is the unsupervised state of the art method for onc-class outlier detection on CIFAR-10 They train their model via contrastive learning, with transformed (shifted) instances of an image itself as additional negatives.
12

---

Published in Transactions on Machine Learning Research (04/2023)
plane
car
deer
frog
horse
PIELEE ENIC
best IN-DIST images
best OOD images
worst IN-DIST images
Figure 8: Visual examples from selected CIFAR-10 classes the IN-DIST and OOD test sets. Every row shows a CTFAR-10 class as IN-DIST. From left. right: the three images with the lowest outlier score of the IN-DIST test set, the three images with the lowest outlier score of the OOD test set and the three images of the IN-DIST test set with the highest outlier score.
Benchmark datasets We conduct experiments using CIFAR-10, CIFAR-100 and CelebA as in- distribution. First, we run experiments on single classes of our datasets and compare test time to the other classes not seen in training. We also conduct experiments in the dataset-vs-dataset setting, where we train on the whole CIFAR-10 dataset and compare to other datasets at test time.
5.3.1 One-vs-rest results
We run experiments on CIFAR-10, CIFAR-100 (Krizhevsky 2009) and celebA (Liu al. 2015). We work in the one vs-rst- setting, where one class is used as inlier data, and all the other classes of the same dataset are used as anomalics at test time. For CIFAR-100, show results for superclasses. For cclebA, divide the dataset into two classes by the given gender attribute and use one of the classes inlier distribution and the other one anomalies.
CIFAR-10 and CIFAR-100 superclasses We evaluate our method on the CIFAR-10 and CIFAR-100 datasets as inlier distribution with the IMAGENET dataset as contrastive distribution. Note that all CIFAR- and -100 classes are subset of the IMAGENET dataset. The discussion Section holds also in practice: The broader contrastive IMAGENET distribution with overlap of the CIFAR inlier distributions does not negatively affect the performance, in contrary method achieves state of the art results almost all classes when evaluated under One-vs-Rest setting. The quantative results of our model shown in figure are sensible from a human perspective: The IN-DIST images with the highest outlier score show differ from typical IN-DIST images and some might be missclassified images of the CIFAR-10 dataset (e-g., the most anomalous frog and horse image in the figure). The OOD with the lowest outlier scores share semantic similaritics with the IN-DIST class, c.g., cars and trucks or dogs and cat. We show quantitative results table One can see that our method one of the best performing models in all settings, while the flow-ratio ablation method struggles in some settings and performs similarly to the simple flow approach. We think the
13

---

Published in Transactions on Machine Learning Research (04/2023)
latter is result of the training the intermediate MoCo feature space. To further investigate method, we show the confusion matrix for all classes in table 3 By looking at the failure cases, we can verify that the model focuses on semantic features: the two worst pairs are deer-horse and cat-dog, which are semantic similar classes. Truck and car have nearly perfect scores; only truck versus car fails (AUROC of about 90). This also shown in figure [ where we show the scores of all CIFAR-10 images for model trained on the deer class as IN-DIST. We conduct the same experiments on the twenty CIFAR-106 superclasses and verify the CIFAR-10 results: The contrastive normalizing flow outperforms or is par with the benchmarks and the baseline methods and reaches state of the art outlier detection performance. Our fine-tuned method CF-FT perforosms the benchmark methods. This advantage is statistically significant, we show table of p-values in appendix A.6 The CIFAR-100 results for all superclasses and methods can be found in table 13 in the appendix.
Table 2: AUROC scores on CIFAR-10 classes for the One- Ose Rest setting. PCA, KDE and KNN are taken from the PyOD library (Zhao et al. 2019a). GOAD from Bergman and Hoshen 2020 CSI from Tack et al. 20200, and Roran(( from Hendrycks et al. (019bb.. MSE, MSE-ratio, and Flow are three ablation methods of our method contrastive normalizing flow (CF). Flow-ratio the 'Hierarchies of Distributions' method by |Schirmeisser et al. (2020), but applied on the MoCo feature space. OE denotes the outlier exposure method by Hendrycks al. (2019a). All methods within one standard deviation of the best AUROC score per class are highlighted. i denotes methods using contrastive dataset.
method
plane
bird
cat
deer
KDE
94.4
99.1
90.4
90.4
93.3
PCA
94.2
98.9
90.6
90.4
93.2
KNN
96.0
98.7
92.3
90.4
93.6
GOAD
75.5
94.1
81.8
72.0
83.7
Rot+T
77.5
96.9
87.3
80.9
92.7
CSI
89.9
99.1
93.1
86.4
93.9
MSE
94.6
99.1
90.4
90.5
93.7
MSE-ratio f
92.8
98.5
89.8
89.7
91.8
Flow
96.1 0.2
97.5 = 0.2
92.6 + 0.2
89.8 + 0.4
93.3 = 0.3
Flow-ratio
95.9 + 0.2
97.7 + 0.4
93.5 + 0.5
90.0 + 0.1
93.2 + 0.5
OE
96.5 + 0.8
99.2 - 0.1
92.9 = 1.7
92.6 - 0.6
93.8 = 0.3
CF (ours) i
96.9 + 0.3
99.0 + 0.1
94.6 + 0.1
92.8 + 0.4
93.5 + 0.4
CF-FT (ours) f
97.4 = 0.2
98.9 + 0.2
94.9 + 0.5
93.5 + 0.6
94.8 + 0.6
method
dog
frog
horse
ship
truck
mean
KDE
92.1
96.2
94.7
98.5
98.2
94.7
PCA
93.0
96.6
95.0
98.6
98.3
94.9
KNN
95.7
98.0
95.8
98.5
97.3
95.6
GOAD
84.4
82.9
93.9
92.9
89.5
85.1
Rot+T
90.2
90.9
96.5
95.2
93.3
90.1
CSI
93.2
95.1
98.7
97.9
95.5
94.3
MSE
91.4
96.3
95.2
98.7
98.2
94.8
MSE-ratio t
92.5
95.4
94.3
98.3
97.6
94.1
Flow
95.7 + 0.4
98.0 + 0.4
94.7 + 0.4
97.8 + 0.0
96.6 + 0.1
95.2 + 0.2
Flow-ratio
95.9 = 0.1
98.2 + 0.0
95.2 + 0.3
97.5 + 0.4
96.6 - 0.5
95.4 + 0.2
OE
93.8 + 0.3
97.6 + 0.4
96.6 0.5
98.4 0.1
98.6 + 0.3
96.0 + 0.2
CF (ours) f
96.1 = 0.4
98.2 + 0.0
06.3022 0.2
98.6 - 0.1
98.5
0.0
96.5 0.1
CF-FT (ours) {
96.2
98.5 + 0.1
96.9 + 0.3
98.7 0.1
98.4 0.3
96.8 0.1
14

---

Published in Transactions on Machine Learning Research (04/2023)
Table 3: Confusion matrix on CIFAR-10 for the contrastive normalizing flow (CF). Every row shows results for model trained on one Cifar-10 class as inlier distribution and evaluated against all other CIFAR-10 classes. Hard cases (AUROC<90) are printed bold.
plane
car
bird
cat
deer
dog
frog
horse
ship
truck
mean
plane 99.6
98.3
96.5
98.7
98.2
99.4
98.1
98.2
88.3
97.2
96.9
car
99.9
99.8
99.9
99.9
99.8
99.8
99.2
93.0
99.0
bird
94.9
99.9
94.6
83.5
97.3
88.8
93.3
99.3
99.8
94.6
cat
97.9
99.7
93.9
88.7
72.3
90.9
92.0
99.1
99.7
92.8
deer
97.7
99.4
92.3
93.8
96.1
93.2
70.5
98.8
99.7
93.5
dog
99.5
99.6
98.3
82.9
95.8
98.6
91.2
99.4
99.8
96.1
frog 98.8
99.7
96.8
95.9
95.9
98.8
99.4
99.1
99.9
98.2
horse 98.6
99.8
97.1
96.4
82.3
94.4
99.0
99.4
99.8
96.3
ship 93.8
98.3
99.6
99.4
99.5
99.6
99.5
99.5
98.4
98.6
truck 98.4
91.2
99.8
99.6
99.7
99.8
99.7
99.5
98.7
98.5
CelebA In the following experiment we want to measure the performance for contrastive distribution dissimilar to the in-distribution data on real world data: We evaluate our model on the CelebA dataset. We use the gender attribute given in the dataset to divide the dataset in two classes, and use each individually as inlier distribution and the other class as test-time outlier distribution. Even though our rastivve distribution for training the contrastive normalizing flow and the flow-ratio method is the IMAGENET dataset, which does not contain close-up pictures of human faces, our method beats the baselines. However, the lacking overlap of the contrastive distribution with the test-time outlier distribution explains why the contrastive approaches such as the flow-ratio method and our contrastive flow do not outperform the other baselines by a clear margin and the overall performance leaves room for improvement. The fine-tuned method CF-FT does not show the same performance improvement in the CIFAR-10 10 and CIFAR-100 experiments. We report the results in table
Table 4: AUROC scores of models trained on the celebA dataset split into the provided gender attribute and evaluated against the other class using contrastive normalizing, CSI, and our ablation methods. MCD and KNN are taken from the PyOD library (Zhao et al. 2019a). All models within one standard deviation of the best performing model are highlighted. We neglect the small standard deviation of the deterministic methods introduced by the MoCo preprocessing. Please note that the model only sees images of the in-distribution class and the unlabeled contrastive IMAGENET dalaset al training Lime.
IN-DIST
MCD
KNN
CSI
Flow
Ratio
OE
CF (our)
CF-FT (our)
female
69.7
74.1
68.3
75.8 = 2.1
76.7 1.8
80.3 1.2
83.8 3.2
77.3 1.5
male
76.6
82.3
79.1
86.7 + 0.8
86.4 + 2.3
88.2 + 1.4
87.1 + 2.6
86.9 + 1.8
average
73.2
78.2
73.7
81.3 = 1.1
81.5 + 1.5
84.3 = 0.9
85.3 + 2.1
82.1 1.2
5.3.2 Dataset-ws-dataset
Next we investigate the sctting of multi-modal in-distribution data. and distant OOD samples. For the dataset-vs-dataset setting, the complete unlabeled dataset is used the inlier distribution at training time. We train contrastive normalizing flow on CIFAR-10 and use CIFAR-100, celebA and SVHN as outlier distributions at test time. We report the AUROC scores in table 5 This setting is well studied, egg,, by Nalisnick et al. (2019a).
15

---

Published in Transactions on Machine Learning Research (04/2023)
Table 5: AUROC scores in the lataset-vs-dataset setting. We train on CIFAR-10 as IN-DIST and IMA-
GENET as contrastive distribution. Additional results are reported in table I0 The standard deviations
are taken over three runs. All models within one standard deviation of the best performing model are high- lighted.
method
CIFAR1O CIFARIO0 CIFARIO SVHN
CIFAR1O vs celebA
mean
MSE
70.0 + 0.0
20.0 + 0.0
92.2 + 0.0
60.7 + 0.0
MSE-Ratiof
74.8 + 0.0
42.0 + 0.0
91.5 = 0.0
69.4 + 0.0
Flow
82.9 + 0.6
65.4 + 3.3
100 + 0.1
82.8 + 1.1
Flow-ratio i
84.7 + 0.3
68.3 + 6.6
100 + 0.0
83.3 + 2.2
OEt CF (ours) f
81.3 + 0.9
65.0 + 2.3
99.9 + 0.1
82.8 + 0.8
84.4 = 0.3
90.3 1.5
99.8 + 0.3
91.5 - 0.5
CF-FT (ours) i
84.7 0.2
84.7 1.4
99.7 + 0.4
89.6 0.5
5.3.3 Discussion of benchmark experiments
We investigate the performance of our difference method in benchmark experiments on CIFAR-10, CIFAR- 100 and CelebA. We find that our model is for almost all settings the best performing or within one standard deviation of the best performing method. We report a confusion matrix in table 3 where we see that most difficult pairs of inlier and outlier data semantically similar classes (e.g., horse-deer or cal-dog). We show in the appendix in section A.10 in-distribution images with high outlier score and out-of-distribution images with low outlier score. We argue that these images also for humans hard to decide and some of them may be missclassified in the original CIFAR-10 dataset. We see for the CIFAR-10 and CIFAR-100 single-class experiments a significant performance gain for our fine-tuned method CF-FT, for the larger and more diverse in-distribution datasets (CelebA and dataset setting) the contrastive flow fully trained with our new objective shows the best performance.
6 Conclusion
We propose a novel method to train density estimators. The method relies on the use of a contrastive dataset for training: The training objective maximizes the log-likelihood in-distribution data while minimizing the log-likelihood the contrastive dataset. We show that using our training objective, the density estimator learns the normalized positive difference of the in-distribution and the contrastive distribution. We use the new objective to train normalizing flows and show its application to outlier detection. To increase the performance of the method and reduce the computational cost, we employ a pretrained feature extractor on which the contrastive normalizing flow operates. We study the influence of the constrastive distribution in several experiments and compare to the ratio method Ren et al. (2019) and the standard normalizing flow without contrastive dataset. We find that our model is robust for sensible choices of the contrastive distribution and alleviates theoretical problems of the ratio method also in the experiments. Furthermore, we show that, method perperfomms state-of-the-art methods from the literature on several benchmark datasets. In this study, utilized the contrastive normalizing field solely for outlier detection. However, contrastive normalizing flows have diverse potential applications such as datasct reduction and lcarning the difference distribution between two datasets. To comprehend the high-dimensional difference distributions, more research necessary.
Acknowledgments
We want to thank all our TMLR reviewers, in particular Reviewer wnbF from the first submission for the comments and questions which helped to significantly strengthen the paper. Furthermore we want to thank Felix Draxler, Dan Zhang and Philipp Geiger for proofreading the manuscript and their helpful suggestions. We would like to acknowlodge the TMLR editorial team for their efficient and professional handling of the submission process.
16

---

Published in Transactions on Machine Learning Research (04/2023)
References
Lynton Ardizzone, Till Bungert, Felix Draxler, Ullrich Köthe, Jakob Kruse, Robert Schmier, and Peter Sorrenson. Framework for Easily Invertible Architectures (FrEIA). Technical report, Computer Vision and Learning Lab, University of Heidelberg, 2018-2023. URL https:
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and UlIrich Köthe. Guided image generation with conditional invertible neural networks. arXiv preprinl arXic:1907. 023992, 2019.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018.
Liron Bergman and Yedid Hoshen. Classification-basee anomaly detection for general data. In International Conference on Learning Representations, 2020.
Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. orXiv preprint arXiv:2002. 10445, 2020.
Terrance E Boult, Steve Cruz, Akshay Raj Dhamija, Manuel Gunther, James Henrydoss, and Walter J Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 9801-90 9807, 2019.
Benamar Bouyeddou, Fouzi Harrou, Ying Sun, and Benamar Kadri. An effective network intrusion detection using hellinger distance- based monitoring mechanism. In 2018 International Conference Applied Smart Systems (ICASS), pages 1 6. IEEE, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607.
PMLR, 2020.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXic:1810.01399 2018.
Matan Jacob Cohen and Shai Avidan. Transformaly-two (feature spaces) better than one. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4060-4069, 2022.
Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint ar xrxi:20005.000 02357, 2020.
Angelo Coluccia, Alessandro D' D'Acooo,o, and Fabio Ricciato. Distribution-lussd anomaly detection in net- work traffic. Data Traffic Monitoring and Analysis: From Measurement, Classification, and Anomaly Detection to Quality of Experience, pages 202 216, 2013.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1 30, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE conference on computer vision and pottern recognition, pages 248-255. Ieee, 2009.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa, Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Rep- resenlations, 2021.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
17

---

Published in Transactions on Machine Learning Research (04/2023)
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705-1714, 2019.
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 98 107, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference computer vision and pattern 2020.
recognition, pages
Douglas Heaven et al. Why deep-learning ais are so easy to fool. Nature, 574(7777):163 2019.
Dan Hendryeks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019a.
Dan Hendrycks, Mantas Mazcika, Saurav Kadavath, and Dawn Song. Using self-supervise learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019b.
Nathalie Japkowicz, Catherine Myers, Mark Gluck, et al. A novelty detection approach classification. In IJCAI, volume 1, pages 118.523. Citeseer, 1995.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Πly Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances neural information processing systems, 29, 2016.
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of- distribution data. Advances in neural information processing systems, 33:20578- -25789, 2020.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(1)):3964 3979, 2020.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pages 3730-3738, 2015.
Yuchen Lu and Peng Xu. Anomaly detection for skin discase images using variational autocncoder. arXiv preprinl X80:01007 2018.
Milad Memarzadeh, Bryan Matthews, and Ilya Avrekh, Unsupervised anomaly detection in flight data using convolutional variational auto-encoder. Aerospace, 7(8):115, 2020.
Fric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Takshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019a.
Eric Nalisnick, Akihiro Matsukawa, Yoe Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXio preprint arxiv: 906. 006.02994, 2019b.
18

---

Published in Transactions on Machine Learning Research (04/2023)
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arxi 1111888777744 2018.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1))2617 2680, 2021a.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617-2688 2021b.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang, Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2898-2999 2019.
Jie Ren, Peter J Lit, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International con- [erence on machine learning, pages 1530 1538. PMLR, 2015.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. To- wards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Reecognition, pages 14318 14328, 2022.
Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semi-supervised defect detection with normalizing flows. In Proceedings of the IEEE/CVF winter conference on applications of compuler vision, pages 1907 1916, 2021.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emnanuel Müller, and Marius Kloft. Deep one-class classification. In International conference machine learning, pages
PMLR, 2018.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Transactions of Machine Learning Research, 2021.
Robin Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. Advances in Neural Information Processing Systems, 33:21038-211 02310104999 2020.
Joan Serrå, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complex- ity and out-of-distribution detection with likelihood-basel generative models. In International Conference on Learning Representations, 2020.
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density-basok weighting for imbalanced regression. Machine Learning, 110:2187 2211, 2021.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11830 11852, 2020.
Hang Wang, David J Miller, and George Kesidis. Anomaly detection of lest-Lime evasion attacks using class-conditional generative adversarial networks. arXiv preprint arXiv:2105.10// 2021.
Frank Wilcoxon. Individual comparisons by ranking methods. Biomelrics, 1 (6):80 83, 1945.
Sangwoong Yoon, Yung-Kyun Noh, and Frank Park. Autoencoding under normalization constraints. International Conference on Machine Learning, pages 12087-12097. PMLR, 2021.
19

---

Published in Transactions on Machine Learning Research (04/2023)
Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows. arviv preprinl arXic:211 1101111.00000 11.006777 2021.
Chunkai Zhang, Shaocong Li, Hongye Zhang, and Yingyang Chen. Velc: A new variational autoencoder based model for time series anomaly detection. arXiv preprint arXim:1907.00 702, 2019.
Lily Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-of-distribution detection with deep generative models. In International Conference on Machine Learning, pages 12427 12436. PMLR, 2021.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20:1 7, 2019a.
Zilong Zhao, Robert Birke, Rui Han, Bogdan Robu, Sara Bouchenak, Sonia Ben Mokhtar, and Lydia Y Chen. Rad: On-line anomaly detection for highly unreliable data. arXiv preprinl arXie: 191 1..4.... 1.04883, 2019b.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International confer- ence learning representations, 2018
20

---

Published in Transactions on Machine Learning Research (04/2023)
A Appendix
A.1 Training Pseudocode
To clarify how the model is trained, we report the training process in pscudocode in algorithm !
Algorithm Training process
With MoCo feature extractor, density estimator Ta and alla (x) = log probability,, (x) for all epochs do for all batches æ inlier data and y ~ contrastive data do for all æ; x. and yjj y do
c = Moooo(;;)....... D; = MoCo(y,)
lossPos; (á;)
lossNeg, = nllg(
lossNog; = clamp(lossNeg,, None, tsh)
end for
loss Σx lossPos: M E, lossNeg,
GradientStep(0, loss)
end for end for
A.2
Implementation details
A.2.1 Contrastive Loss Clamping
For our loss, we apply small deviation from the theory: The second term in equation results in the theoretical optimum pa(x) Va supp(q > p), but the loss diverges because lim log x is unbounded. To handle this mismatch, we clamp log p(x) at a threshold as a lower bound on the likelihood. Therefore the objective leads to logp(æ) < Væ E supp(g q > p). which we find to be sufficient for successful outlier detection. For our experiments, we found e 0 sufficient for a good outlier detection performance. For unknown datasets, we recommend to train a standard normalizing flow first and using the distribution of the in-distribution likelihoods to determine a constant clamping parameter, c.g. the 10% quantile of the in- distribution log density plus fixed offset (e-g., In(10) 2.3) We investigate the influence of the threshold in the appendix in A.4 and find that the outlier detection performance constant under reasonable changes of e.
A.2.2 MoCo Finetuning
For datasets with image sizes significantly smaller than the images used in the MoCo implementation with 224 by 224 pixels, the features dominated by upsampling artefacts and are highly correlated between different images. To reduce this problem without changing the setup, finetuned MoCo trained on 244*244 images on smaller images by shrinking IMAGENET images to 32 by 32 pixels and then using upsamling to 224 by 224 pixels again. We discuss this in appendix A.4
A.2.3 Hyperparameter settings
As our contrastive normalizing flow, standard normalizing flow and the ratio method use the same archi- tecture and similar training procedures, decided to standard architecture similiar to the architecture proposed in https:://viii http://vveleaannnn github.io//rEIA. wwt.Fr//////////// bubbuullddi Attp/EEiA/.buld//tmmlll. httta///tttorraalaaaaaaeee////// cow.ully_ooneeted..... html We used the AUROC score between rn-distribution validation set and imagenel validation sel (con- trastive) as proxy for the outlier performance and tuned our method and all our ablation methods. For our contrastive normalizing flow, the reported hyperparameter worked best over all real world data experiments. All tested models showed to be robust on small hyperparameter changes. We also used this proxy AUROC to implement early stopping and prevent overfitting.
21

---

Published in Transactions on Machine Learning Research (04/2023)
Table 6: Hyperparameter settings for all contrastive normalizing flow experiments. CF stands for our contrastive normalizing flow model.
Hyperparameter
Batchsize
256
Optimizer
Adam
Learning rate
10-3
CF implementation
Freia (Ardizzone et al. 2018-2023)
CF architecture
8 Freia AlllnOneelocks
CF subnetwork
ML. with 512 hidden layers and ReLU act.
CF affinc clamping
3.
CF dimensions
128 (MoCo output)
Contrastive loss clamping
0.
MoCo implementation ion MoCo finetuning
He el al. (2020)
Added 32*32 downsamp ing to preprocessing
Table 7: AUROC for different methods of the PyOD (Zhao et al. 2019a) library on the CIFAR-10 10 one-vs- rest setting with standard deviation over three runs. The first column shows the results on the image data, the second on unnormalized MoCo features and the last on normalized MoCo features. Using normalized MoCo features gives the best results for most. methods. The three best performing methods KDE, PCA and KNN were reported in table 2 The small standard deviations for the deterministic models (COPOD, LOF, MCD, ABOD, KDE, PCA and KNN) introduced by the MoCo preprocessing and upsampling. They are reduced by the normalization, such that the rounded values are 0.0 for the last. four methods.
Method
No Feature extractor
FE + No Normalization
FE + Normalization
COPÓD
54.8 = 0.0
63.0 = 0.1
86.3 = 0.1
INNE
57.6 + 0.7
77.9 + 0.7
88.6 = 0.6
LOF
57.5 = 0.0
94.2 + 0.1
89.9 = 0.1
MCD
92.6 0.0
91.6 + 0.1
iForest
59.8 + 0.2
79.5 0.6
92.2 = 0.2
Sampling
58.8 = 1.2
82.0 1.6
93.5 0.3
ABOD
58.7 = 0.0
92.9 0.1
94.0 0.0
KDE
54.3 + 0.0
77.3 0.1
94.7 - 0.0
PCA
58.7 + 0.0
79.4 + 0.1
94.9 + 0.0
KNN
59.1 - 0.0
91.5 0.1
95.6 = 0.0
CF (ours)
95.8 + 0.2
96.5 - 0.1
A.3 PyOD Experiments
We rum additional experiments for methods from the PyOD library (Zhao et al. 2019a) without feature extractor and with feature extractor with and without normalization of the features (the results in the main paper are with feature extractor and normalization). The results are reported in table 7 g 9 and I0 The small standard deviations for the deterministic models (COPOD, LOF, MCD, ABOD, KDE, PCA and KNN) are introduced by the MoCo preprocessing and upsampling. The experiments show that using the MoCo feature space significantly improves the outlier detection performance for all models. Normalizing the Teatures improves the score for most models.
A.4 Ablation studies
Tuning the Threshold c To show the dependence of our model on the clamping threshold c, we run two ablation studies: First, we learn one-dimensional contrastive normalizing flows on the experimental setup of our toy experiment in section 4 As we know the ground truth difference, we choose different thresholds
22

---

Published in Transactions on Machine Learning Research (04/2023)
Table 8: AUROC for different methods of the PyOD (Zhao et al. 2019a library on the CIFAR-100 one-vs- rest setting with standard deviation over three runs. The first column shows the results on the image data, the second on unnormalized MoCo features and the last. on normalized MoCo features. Using normalized MoCo features gives the best results most methods. The three best performing methods KDE, PCA and KNN reported in table 2 The small standard deviations for the deterministic models (COPOD, LOF, MCD, ABOD, KDE, PCA and KNNN are introduced by the MoCo preprocessing and upsampling. They are reduced by the normalization, such that the rounded values are 0.0 for the last three methods.
Method
No Feature Extractor
FE + No Normalization
FE + Normalization
COPOD
53.0 + 0.0
54.0 + 0.1
73.4 + 0.2
INNE
57.2 = 0.7
69.9 + 0.4
83.3 + 1.0
LOF
59.9 + 0.0
90.1 + 0.1
87.2 + 0.1
iForest
59.5 = 0.3
71.2 = 0.4
89.3 + 0.2
Sampling
58.4 0.4
77.1 + 0.7
91.2 + 0.5
MCD
91.5 + 0.0
91.5 + 0.2
ABOD
60.2 + 0.0
89.9 + 0.1
92.8 + 0.1
KDE
54.0 = 0.0
67.9 0.0
93.0 + 0.0
PCA
57.0 = 0.0
70.5 0.1
93.2 + 0.0
KNN
60.0 + 0.0
86.4 + 0.0
94.8 + 0.0
CF (ours)
93.8 + 0.2
95.1 + 0.1
Table 9: AUROC for different methods of the PyOD Zhao et al. 2019a) library on the celebA class-vs-class setting with standard deviation over three runs. The first column shows the results on the unnormalized MoCo features and the second normalized MoCo features.
FE + No Normalization
FE + Normalization
Method
male
female
male
female
LOF
67.6
0.5
61.6 0.1
53.7 0.3
55.1 0.1
ABOD
82.1
0.3
63.7 0.1
68.9 + 0.4
76.4 0.1
MCD
79.0 - 0.2
67.6 + 0.0
69.7 0.0
76.6 - 0.2
LODA
74.8 + 2.9
57.9 + 5.8
81.2 + 1.5
77.9 + 4.0
KNN
77.2 + 0.1
70.1 + 0.2
74.0 + 0.1
82.3 + 0.0
Table 10: AUROC for different methods of the PyOD ZZao et al. 2019a) library on the CIFAR-10 vs. dataset setting with standard deviation over three runs. The first column shows the results on the unnor- malized MoCo features and the second on normalized MoCo features.
FE
No Normalization
FE + Normalization
Method
CIFAR-100
SVHN
cclebA
CIFAR-100
SVHN
celebA
KNN
76.4 0.1
29.5 0.6
98.91
0.0
83.8 0.1
89.6 0.3
99.9 0.0
LOF
81.4 0.0
44.2 + 2.6
99.8 + 0.2
71.3 + 0.0
33.7 1.2
95.3 0.2
iForest.
52.0 + 1.0
8.5 4.5+00 0.0
70.4 0.4
68.6 + 0.7
56.0 + 10.4
65.9 + 2.5
LODA
51.2 + 2.5
12.2 + 2.8
61.3 + 4.7
00.5
51.8 3.2
81.2 + 8.7
MCD
71.8 + 2.5
66.0 + 1.9
97.1 + 2.2
64.5 1.5+11 7.7
74.5 = 12.0
83.4 + 17.0
23

---

Published in Transactions on Machine Learning Research (04/2023)
10:
0.5
5 8.0
32.0
0.4
ground truth
10:7
10
log density
10:43
10
10
3
Figure 9: Density plots for models trained on samples x ~ N(, 1) as in-distribution and y -- q- (1, as contrastive dataset for different thresholds c. On the right, the densities and thresholds are plotted log space. Because 0.0 is larger than the in-distribution log density, the contrastive dataset is ignored while training and the in-distribution density is learned. For large the approximation of the in-distribution suffers.
threshold 0.0
threshold 1.0
out-of-distribution
out-of-distribution in-distribution
in-distribution
0.5
0.0
threshold E 5.0
threshold
10.0
1.0
1.0
out-of-distribution
out-of-distribution In-distribution
In-distribution
0.5
0.5
0.0
0.0
-10
5
10
20
-10
-5
5
20
Figure 10: Histograms for in-distribution and out-of-distribution outlier scores for contrastive normalizing flows trained the CIFAR-10 10 cat class as in-distribution, with different thresholds c. The AUROC score is constant over all models (92.8 + 0.4)
ranging from values larger than the ground truth log liko-likelihood (c= 0.0) values orders of magnitude smaller (e = 100.0). We show the learned densities of our models in figure As one can see, the models approximate the ground truth density. If the thresholds gets too small, contrastive data is used in the objective and the in-distribution density is learned. If the threshold is too large, the approxination for high likelihoods worsens as the model needs to model the tails of the distribution.
We also run an ablation study on the cat class of the CIFAR-10 dataset. We plot in figure 10 histograms for the in-distribution and out-of-distribution (all other CIFAR-10-classes) outlier scores for models trained with different thresholds e. As simple ood-cases are pushed above the threshold, but hard ood-caser overlap with the in-distribution scores for all thresholds, the performance stays constant for all four models.
24

---

Published in Transactions on Machine Learning Research (04/2023)
IN-DIST Flow (inceptionV3)
Ratio (inceptionV3)
CF (ours, inceptionV3)
CF (ours, MoCo)
male
74.4 + 1.1
75.1 2.7
78.2 3.8
83.8 + 3.2
female
82.0 + 2.4
82.0 2.8
83.1 2.4
87.1 + 2.6
average
78.7 + 1.3
78.6 2.0
80.7 2.2
85.3 + 2.1
Table 11: Performance of our model and ablation methods on the (projected) inceptionV3 feature space.
Table 12: Ablation for normalization of the MoCo features. Normalization the features improves clustering, as one can see by the improvement on our MSE baseline.
Without Norm.
With Norm. MSE
dataset
MSE
CF
CF
CTFAR-10
77.1
95.8
94.3
96.5
CIFAR-100
67.6
93.8
92.2
95.1
No normalization of the features Because of the cosine similarity in the contrastive learning objective, the contrastive loss is invariant under changes of the norm of the feature vectors. For the normalizing flow, the norm of the input vector has an important influence on the likelihood; therefore we decided to normalize all feature vectors to account for this mismatch. While this changes the topology the feature space drastically, the normalization improves the performance of our method. The results can be found table We show the performance for the contrastive normalizing flow and the MSE ablation method because it directly shows the clustering of the features.
12
MoCo finetuning The original MoCo trained on IMAGENET, with image input sizes of 244*244 pixels. When using the feature extractor on smaller images like the CIFAR-10 CIFAR-100 dataset, the MoCo features dominated by upsampling artifacts: The feature dimensions of upsampled images have strong correlation and are mostly independent of the semantic content. To improve the performance of the model, we finetuned the feature extractor with additional MoCo training on first down- and then upsampled IMAGENET data. The resulting features are less correlated and the performance of models trained on the resulting feature spaces is significantly improved. Examples for both feature spaces are shown figure
Use of a different feature extractor We use the inceptionV3 feature extractor, used for the calculation of FID-scores, to investigate the dependence of our model on the MoCo feature extractor used in all our experiments. As inceptionV3 trained as a classifier, we use the penultimate layer for feature extraction. We use the celebA dataset for comparison, the inceptionV3 model only works for inputs of size 299 by 299 pixels or larger. To use the same architocture as for our MoCo experiments, we project the 2048 dimensional inceptionV3 feature space 128 dimensions by summing over fixed groups of 16 features. We show the results in table 11 and compare to the performance on the MoCo feature space. The performance drops for all methods by about 4 AUROC compared to the experiments on the MoCo feature space. This experiment demonstrates that the advantage of our method is not feature space dependent and that the MoCo feature extractor is a good choice for methods.
A.5 CIFAR-100 results and standard deviations
We report in table 13 the AUROCS for all CIFAR-100 clases for all methods and in table 14 the standard deviations for our methods.
A.6 Significance tests
Following the procedure in Demsar 2006). we test the significance of our method against the benchmark methods with the Wilcoxon signed ranks test [Wilcoxon] 1945). We use the means over three runs to compare the different methods. The p-values for the null-hypothesis "our model does not outperform the benchmark method shown in figure 12 (CTFAR-10) and figure 13 (CIFAR-100). The fine-tuned methods outperforms
25

---

Published in Transactions on Machine Learning Research (04/2023)
Original MOCO feature space
Class 3
Class 3
Class
Class 9
0.010
0.005
0.000
-0.005
-0.010
Finetuned MOCO feature space
0.2
0.1
0.0
-0.1
-0.2
0 20
40
60
80
100
120
Figure 11: Qualitative comparison of the original and finetuned MoCo feature space. In the original MoCo feature space, the dimensions have a strong correlation for images different classes, while in the finetuned MoCo feature space the feature dimensions are correlated across different classes.
26

---

Published in Transactions on Machine Learning Research (04/2023)
Table 13: AUROC scores on CIFAR-100 superclasses for the One-Vs-Ress setting. Our method contrastive normalizing flow (CF) hast the best outlier detection performance over all classes. Best AUROC scores per class are bold. i denotes methods using contrastive dataset.
method
KDE
90.7
91.0
96.4
95.2
95.7
90.9
96.4
92.7
92.6
96.5
PCA
91.2
91.0
96.6
95.9
95.6
90.9
96.1
93.2
92.9
96.7
KNN
91.6
92.7
96.3
97.1
97.2
95.4
97.2
94.5
95.6
96.5
CSI
86.3
84.8
88.9
85.7
93.7
81.9
91.8
83.9
91.6
95.0
MSE
90.0
90.5
96.4
94.5
94.9
89.3
96.0
92.0
91.7
96.0
MSE-ratio
90.5
89.5
95.8
95.3
94.7
90.6
95.9
92.6
93.4
96.5
Flow
91.7
92.7
91.7
92.7
91.7
92.7
91.7
92.7
91.7
92.7
Flow-ratio
93.1
92.9
96.0
96.5
97.0
94.5
97.2
94.5
95.1
95.9
OE
93.3
94.2
96.8
95.5
94.9
95.3
97.2
CF (ours)
92.6
94.1
97.3
95.9
96.8
95.0
97.3
94.3
95.0
97.1
CF-FT (ours)
93.8
94.9
97.9
97.4
97.6
95.3
95.8
97.8
method
10
11
12
13
14
15
16
17
18
19
KDE
96.1
92.0
88.7
88.3
96.7
88.8
85.9
97.8
96.4
91.5
93.0
PCA
96.3
92.0
89.5
88.0
96.7
88.6
86.9
97.8
96.0
91.3
93.2
KNN
97.4
94.6
92.3
90.6
98.5
88.8
90.7
97.8
97.3
94.0
94.8
CSI
94.0
95.9
95.1
MSE
95.8
90.7
88.0
87.0
95.5
87.0
85.5
97.6
95.2
89.4
92.2
MSE-ratio
95.3
92.7
88.8
87.9
96.7
96.1
95.9
90.2
92.3
Flow
96.3
93.2
90.1
91.9
97.9
88.8
90.3
97.4
96.7
94.7
93.0
Flow-ratiof
93.6
91.7
92.6
98.0
96.8
95.0
94.6
OE
97.4
94.2
92.0
92.0
98.4
91.5
91.3
98.4
97.0
95.2
95.0
CF (ours)t
97.0
94.5
91.4
91.7
97.9
90.9
91.4
98.4
96.1
95.6
95.0
CF-FT (ours)t
97.1
94.4
92.4
91.7
98.6
92.0
92.2
98.5
97.2
96.5
95.7
Standard deviations over three runs for our method Contrastive normalizing flow (CF) and the bascline methods 'Flow' and 'Flow-ratio' for all CIFAR-100 classes.
Table 14:
class
Flow
Flow-ratio
CF (ours)
CF-FT (ours)
91.7 0.3
93.1 + 0.6
92.6 + 0.4
93.8 + 0.0
92.7 + 0.9
92.9 0.9
94.1 + 0.2
94.9 + 0.4
91.7 + 0.3
96.0 + 0.1
97.3 + 0.2
97.9 + 0.0
3
92.7 0.4
96.5 1.0
95.9 0.5
07.10...
4
91.7 + 0.5
9
96.8 + 0.2
97.4 +7.40..4 0.4
5
92.7 0.4
94.5
95.0 1.0 0.1
9..4
6
91.7 0.2
97.2
97.3 + 0.6
97.6
92.7 0.2
94.5 0.5
94.3 - 0.2
95.3 = 0.4
91.7 0.3
95.1 0.8
95.0 + 0.1
95.8 + 0.1
92.7
95.9 + 0.3
97.1 + 0.6
97.8 + 0.3
10
96.3 0.6
96.1 + 0.6
97.0 + 0.6
97.1 0.1
11
03.2
93.6 0.7
94.5 + 0.2
94.4 0.4
12
00.1
89.7
91.4 0.6
02.4....
13
91.9 + 0.4
91.7 0.5
91.7 + 0.7
91.7 0.3
97.9 1.9 0.5
92.6 0.5
97.9 - 0.7
98.6 0.4
88.8 0.7
89.1 + 0.2
90.9 + 0.5
92.0 0.3
16
9.9
91.0 0.5
91.4 0.5
92.2 0.3
17
97.4 + 0.3
98.0 + 0.2
98.4 + 0.1
98.5 0.2
18
96.7 0.3
96.8 0.5
96.1 + 0.6
97.2
19
94.7 + 0.4
95.0 - 0.3
95.6 - 0.5
96.5 0.1
mean
93.0 + 0.2
94.6 + 0.2
95.0 + 0.1
95.7 + 0.1
27

---

Published in Transactions on Machine Learning Research (04/2023)
p-values the signed
5
0,0025
0.00069
0.000/9
0.000/1
C-F
0.00008
0.0025
0.00049
0.00049
0.00049
0.0049
3.0035
0.00049
0.0004g
0.00040
0.0034
0.02
GDAD
Row
Rew-ratic
p-values for mull-hypothesis 'our model does not outperform the benchmark method" for all methods on CIFAR-10
Figure 12:
p p-values for the Wilecsson signed ranis test or the CIRAR results
0.049
0.073
45
3,0002
0040
0,024
0.006
0.24
0.55
5
5e-07
Sm.07
0.00774
a S--07
Se-07
Se-07
Qm-06
6--05
R Adw-15
KDE
PCA
MSE
MSE-rato
Flw
CE
Figure p-values for null-hypothesis 'our model does not outperform the benchmark method' for all methods on CIFAR-100
all baseline methods significantly for a significance level of 0.05. For CelebA and the dataset setting, the number of samples (n=2 and n=3) is too small for the wilcoxon test. However, in the dataset- seaaset-ssdataas dataset sctting, methods outperform the basclines by clear margin of more than two standard deviations.
A.7 Compute resources
Our experiments were performed on GPU cluster using one cluster node with one Nvidia V100-32GB GPU and 2 CPU using 16 GB RAM. training run our contrastive normalizing flow took approximately 30 minutes. In this publication we show the results of about 200 training runs of the contrastive normalizing flow. Not included in this estimate are baseline method runs and research runs. Our organization is carbon neutral that all its activities including research activities do not leave carbon footprint. This includes our GPU clusters on which the experiments have been performed.
A.8 Beyond the Image Domain
To further investigate our method we apply the contrastive normalizing flow to another data modality: We use the credit card fraud dataset from httpps:/www kaggle www.coo//aaa/o saaatseet/mmg--ie. and apply our contrastive normalizing flow. As contrastive distribution we use the uncorrelated marginal distributions by permuting all samples in batch per feature dimension. We show the results in figure 14 The contrastive normalizing flow and the standard normalizing flow perform par (AUROC of 97.5), where the ratio method fails for outlier detection (AUROC of 41). Our contrastive normalizing flow does not outperform the standard normalizing flow, which is not surprising: KKrichenko et al. (2020) show that low-dimensional image features dominate the likelihood of the normalizing flow and thus lead to problems on image data. This behaviour not expected on tabular data, where the standard normalizing flow is already a good outlier detection method, We think that the contrastive normalizing flow could be useful for other applications in the tabular and other domains: As shown in our ablation study in 5.2.3 the contrastive normalizing flow can be used for outlier detection with known anomalies.
sasttttttssmiguul///reeeeeee wulpuuubereereee tcardfraud,
A.8.1 Discussion
For non-image settings, where no broad contrastive datasel like IMAGENET is readily available, the best choice a set known outliers. necessary, this outlier dataset can be enlarged by data augmentation. If no outliers are known at training time, a potential fallback is to construct the contrastive distribution as a transformed inlier distribution, e.g., by adding Gaussian noise or using the product of the marginal distributions removing all correlations. However, found in the tabular data experiment in section A.8
28

---

Published in Transactions on Machine Learning Research (04/2023)
Standard Normalizing Flow
Ratio Method
Contrastive Normalizing Flow
2.00
train
train
train
ood
ood
ood
1.50
1.25
1.00
0.50
0.25
0.00
.8.
10
10
15
0
15
Figure 14: Standard Normalizing Flow, Ratio method and Contrastive Normalizing Flow applied tab- ular fraud detection dataset. While the ratio method completely fails in the tabular domain, the cantrastive normalizing flow performs par with the standard normalizing flow. We think that the contrastive nor- malizing flow can be used beyond the scope of image outlier detection in various data domains.
Supervised anomaly detection performance
Performance for contrastive dataset
1.0
1.0
std
AUROC score horse class
0.9
ratio
0.9
AUROC score for 000 class
0.8
0.7
0.7
0.5
0.5
contr
std
0.4
racio
0.4
0.0
0.2
0.4
1.0
0.2
0.6
0.8 1.0
contrastive distr. mixture fraction vs.
contrastive distr. mixture fraction IMAGENET
Figure 15: Training directly on images without feature extractor: Performance for our model, the ratio method and a standard normalizing flow trained on the CIFAR-10 deer class as in-distribution and a mixture of IMAGENET and the horse class of CIFAR-10 as contrastive dataset. On the left we plot the AUROC against the horse class included in the mixture (which correspond to fully supervised setting when / = 0.0), the right against all other CIFAR-10 classes (without horse and deer). Standard deviations are computed over three different seeds. Please note that although both graphs use the same trained model, the experimental setting is vastly different: On the left the distribution of the outliers is already known at training time, where on the right we stay in the uniformed outlier detection sctting.
that this did not improve performance compared to a standard normalizing flow. We leave the choice of the contrastive distribution in the challenging setting without known outliers or broad contrastive distribution for future work.
A.9 Applying contrastive normalizing flows on images without feature extractor
In this soction, we applied the contrastive normalizing flow directly on image data, utilizing the GLOW pytorch implementation from https://githhti com.rrosiaaltyygloooyoooooo with the same experimental setup ablation studies in section 5.2 Due to longer compute times, we only conducted experiments
29

---

Published in Transactions on Machine Learning Research (04/2023)
OOD performance on difficult cases
OOD performance on easier cases
contr
std
1.0
AUROC Score for hard OOD data
AUROC score for ood data
0.7
0.7
0.6
0.6
0.5
0.5
contr
0.4
0.4
ratio
0.2 0.4
0.4
0.6
0.8
contrastive distr. mixture fraction of IMAGENET vs. in-distribution data
contrastive distr. mixture fraction F of IMAGENET vs. in-distribution data
Figure 16: Training directly on images without feature extractor: Performance for our model, standard normalizing flow and the ratio method trained the CIFAR-10 horse class as in-distribution and of mixture of IMAGENET and in-distribution data as contrastive dataset. denotes the fraction of imagenet samples in the contrastive dataset, so for = 1 the contrastive dataset consists only of imagenet samples and for = 0 the contrastive dataset contains only samples from the CIFAR-10 horse class. On the right we plot the AUROC score against difficult distr--distribttion cases (deer class of CIFAR-10), on the left against all other CIFAR-10 classes (without horse and doer). Errors are computed over three different scods.
lower resolution of (in steps of 0.2). The results shown in Figures 15 and 16 support our findings in section using a MoCo feature extractor: The contrastive normalizing flow reliably outperforms the ratio method and the slandard normalizing flow. In the informed case, the performance of the cont rastive flow without use of a feature extractor is superior to using MoCo, which supports our finding the paper that the MoCo features have some overlap for semantically similar classes. In the uninformed cases, the performance of all three methods drops compared to the use of the MoCo feature extractor. As we are primarily interested in the uninformed outlier detection setting, the results confirm choice of the MoCo feature extractor. This performance improvement can also be observed for standard outlier detection methods reported in table 8.
5.2
However, there are still several open questions and areas for improvement when employing contrastive nor- malizing flow on image data, e.g., sampling from the difference distribution trained on full images produces NaNs in the backward pass, which still needs to be investigated. is possible that the training process re- quires further refinement. and hyperparameter luning, or we need a better underslanding of bigh-dinensional (difference) distributions. Additionally, the contrastive normalizing flow could benefit from architectural im- provements, we observed that training using GLOW architecture full images was unstable in comparison to training on MoCo features. In conclusion, further research is necessary to effectively apply rastivstive normalizing flows directly on image data for outlier detection and other applications.
A.10 Qualitative examples for inlier and anomalies
We show additional visual examples from the IN-DIST and OOD test set. IN-DIST test images with low outlier score shown in figure 17 IN-DIST test images with high outlier score in figure 18 and OOD
test images with low outlier score shown in figure 19
A.11 Receiver Operating Characteristic
To further illustrate our method, we investigate ROC curves for the CIFAR-10 doer class. We use the same experimental setting as in figure the deer class of CIFAR-10 used as inlier distribution and IMAGENET is used for all contrastive methods contrastive dataset. To the performance for two corner cases,
30

---

Published in Transactions on Machine Learning Research (04/2023)
plane
car
bird
cat
deer
dog
frog
ship
truck
Figure 17: Examples of images with low outlier score of the IN-DIST test set for all CIFAR-10 classes. Every
row shows examples for a model trained on one CIFAR-10 class as IN-DIST.
31

---

Published in Transactions on Machine Learning Research (04/2023)
plane
car-
bird
cat
deer
dog
frog
ship
truck
Figure 18: Examples of images with high outlier score of the IN-DIST test. set for all CIFAR-10 classes. Every row shows examples for a model trained on one CIFAR-10 class as IN-DIST.
32

---

Published in Transactions on Machine Learning Research (04/2023)
plane
car
bird
cat
deer
dog
frog
horse
ship
truck
Figure 19: Examples of images with low outlier score of the OOD test set for all CIFAR-10 classes. Every row shows examples for a model trained on one CIFAR-10 class as IN-DIST.
33

---

Published in Transactions on Machine Learning Research (04/2023)
ROC for PyOD-methods for easy 0OD
ROC for Flow-methods for easy OOD
0.6
(ours)
COPOD
cfft (ours)
0.2
KDE
rallo method
normalizing flow
MCD
MSE
COF
MSE-ratio
ROC for PyOD- methhds for difficult GOD
ROC for Flow-methods difficult OOD
KNN
COPOD
KDE
MCD
0.6
(ours)
(ours)
ratio method
normalizing MSE
0.0
0.4
0.0
0.6
0.8
Figure 20: ROC curves for the CIFAR-10 deer class as in-distribution, IMAGENET as contrastive dataset and the horse class as difficult OOD cases and the truck class as easy OOD cases. We plot MSF and MSF- ratio with the flow models. We see that our contrastive normalizing flow method has an advantage over all other methods for low outlier scores in the case of difficult OOD samples. The fine-tuned contrastive normalizing flow improves for allmost all scores.
two CIFAR-10 classes separately anomalies: Our model showed for the pair of deer and horse class the worst performance over all CIFAR-10 pairs (when trained deer inlier), even in the supervises case (section 5.2.3) our model and the likelihood ratio model reach only an AUROC of 95. In contrast, for the doer class as inlier and truck class as outlier, our model shows almost perfect separation. See table for the confusion matrix over all classes. The ROC curves for both outlier classes and various methods are shown figure 20 When plotting the outlier detection performance for the horse class of CIFAR-10 (hard outliers), our contrastive normalizing flow (cnf) and the fine-tuned normalizing flow (cf-ft) show the best performance with an AUROC of 73 respectively 75. For the truck class (casy outliers) almost all models can separate nearly perfectly wetween inliers and anomalies.
34


Reviews:
[{"review_id": "TJ6tMiKrIm", "reviewer": "Reviewer_ifF9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I reviewed the prior version of this work so my comments only address the main changes. Overall, the additions and explanations improved the paper significantly.\n\n*Comments on updates*\n- The fundamental idea that ratios are unstable when learned from samples---particularly when outliers of both the original and contrastive distributions---compared to differences is very important. I would suggest highlighting this idea more near the end of section 4. Perhaps provide an enumerated list of the key benefits and distinctions from likelihood ratio or likelihood.\n\n- I appreciated the explanation and logical flow of why use a feature extractor in this case.\n\n- I also appreciated the comparison to other feature extractors and at least some justification for MoCo, both in words and in new experiments.\n\n*Questions*\n- Why are there two modes in Figure 1 (right)? Is there a good explanation for this?\n\n- Is the positive difference distribution useful for anything beyond outlier detection? At this point, it only seems like outlier detection, but could it be useful more broadly?\n\n\n*Minor Comments*\n- You should point out the $\\mu = 0$ case earlier and why normalizing flows work better in this case. Also, the use of $\\mu$ is somewhat confusing. To double check, does $\\mu=1$ mean that all the contrastive examples are from ImageNet? Perhaps clarifying the wording could help as it took me several reads to understand the figure.  Perhaps just something like \"% Contrastive that is in-distribution\".\n\n- \"semantic distant classes\" -> \"semantically distant classes\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an alternative objective for learning a flow model based on a target in-distribution dataset and a contrastive dataset.\nIn contrast to likelihood ratios, this method learns a positive difference distribution (where the difference is w.r.t. to the PDFs).\nThis method seems to be more stable than likelihood ratios or simple likelihood, and is experimentally shown to be competitive or outperform standard baselines.\nMore broadly, the training of a positive difference distribution may be interesting, though the paper does not discuss this.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I suggest changing the title from \"based\" to \"for\". It makes more sense as \"based\" might require hyphens like \"model-based\", i.e., \"Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data\".\n", "broader_impact_concerns": "No broader impacts section but it does not seem required for this work.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aoX9Dq34o1", "reviewer": "Reviewer_GvHC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n1. Simple method that works well in practice\n2. Clear exposition of the CF based methods, with intuitive explanations on why they might be better than more traditional distribution ratios\n3. Quite extensive experiments that provide adequate evidence for the claims and performance of the methods\n\n*Weaknesses*\n1. The method relies on a pre-trained feature extractor and results can degrade depending on its choice (c.f. Table 10 at the appendix)\n2. Optimization is fundamentally unstable and the authors are clamping the log probabilities during training to avoid the instabilities. The amount of clamping is an additional hyperparameter that needs to be tuned. Similar clamping can also be applied to the baselines, (to e.g., avoid the instabilities when doing the ratio $p(x)/q(x)$), when e.g., one is learning $\\log p(x)$ and $\\log q(x)$. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes contrastive normalising flow (CF) for the task of outlier detection.  CF is a method that learns to model the difference of two densities, $p(x)$, (the inlier distribution) and $q(x)$ (the contrastive / negative sample distribution) on areas where $p(x) > q(x)$. By using the log probability under the learned density model as an outlier score, outliers occur when a) samples have higher density in $q(x)$ rather than $p(x)$, i.e., they come with high probability from the contrastive distribution and b) samples have low probability in both $p(x)$ and $q(x)$, i.e., they are overall unknown. The authors prove that learning this positive difference distribution amounts to training a density model on a simple objective that maximises the log probability of samples coming from $p(x)$ while minimising the log probability of samples coming from $q(x)$. As the method critically relies on evaluations of the log probabilities of samples, the authors employ a normalising flow for the density model which permits exact evaluation of the log probabilities. Furthermore, it should be mentioned that the authors perform density estimation on the feature space of a pre-trained feature extractor, rather the original input space directly. \n\nBesides CF, the authors also propose CF-FT, where the normalising flow is firstly trained to maximise the log probability of samples from $p(x)$ and then a short fine-tuning procedure is done with the CF objective to learn to subtract the appropriate density from $p(x)$ according to $q(x)$.\n\nBoth methods are evaluated in a quite extensively on a variety of settings. Firstly, the authors show on toy tasks that CF is learning the correct difference density and how its differences with more traditional ratio based approaches manifest. Secondly, the authors experiment on outlier detection using a wide variety of inlier, $p(x)$ and outlier / contrastive, $q(x)$, distributions and show how different choices for these affect the outlier detection performance. Finally, they also perform several benchmark tasks where it is shown that both CF and CF-TF perform favourably against the current state-of-the-art.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This work is generally solid and I am willing to recommend acceptance. I only have a few comments that can potentially further improve it (based on the aforementioned weaknesses):\n\na) It would be nice if the authors can show performance when training the densities directly on the input space. This would disentangle a bit the dependence on the pretrained feature extractor. \n\nb) In the CF type of approaches the density is trained under a specific lower bound to the log-likelihood (the $\\varepsilon$ parameter). It is unclear whether such a lower bound would also help the baselines, e.g., the single normalizing flow for $p(x)$ or the ratio based method of $p(x)/q(x)$.\n\nOther than that,  there are a few typos / places where the explanation can be improved\n\n- This might be personal preference, but “outlier score” is a bit counterintuitive for the $\\log p(x)$ scores; you want a low $\\log p(x)$ score to identify an outlier, so $\\log p(x)$ is more of an inlier score. Alternative, $-\\log p(x)$ could be treated as an outlier score. \n- Figure 1, CF plot; you need to mention the $p(x) > q(x)$, as otherwise the term inside the logarithm can be negative. \n- Page 4, section 3.1: it is mentioned that that NF maps the data to a normal distribution. While this is usually the case in practice, in theory it can be an arbitrary base distribution, so I would rephrase it to be more general. \n- Eq. 1 is incomplete, as for equality one also needs the constant terms of the standard normal base density. It would also be better if the notation is changed to $T_\\theta()$ to show the explicit dependence of $T$ on $\\theta$.\n- Eq. 2, the temperature $\\tau$ needs to be inside the exponent on the enumerator and a parenthesis is needed in the denominator. If $\\tau$ is outside the exponent it cancels out. \n- Underneath Eq. 2 it is mentioned that infoNCE maximises the mutual information; technically,  it maximises a lower bound to the mutual information and it is unclear how tight this bound is.\n- Page 6: a $\\hat{\\theta}$ is introduced on the second paragraph after Eq. 6; it is unclear what it is.\n- $p(x) - q(x) = p(f|s)(p(s) - q(s))$: there is a mismatch between the r.h.s. and the l.h.s. as $x$ is not present. It would be better if it is written as $p(x) - q(x) = \\int p(x|f, s) p(f|s)(p(s) - q(s))df ds$.\n- Third line from the end of page 6: contrastive is mentioned twice in sequence.\n- Page 11, third line of the “How to choose a good contrastive dataset” paragraph: ninty -> ninety.\n- Page 14, first sentence after Table 2: by -> be.\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received two positive reviews:

1. Reviewer ifF9, who also reviewed the previous version of the paper, is satisfied with the improvements in this version.
2. Reviewer GvHC, who reviewed the paper for the first time, is satisfied with the clarity of the exposition and the experimental evaluation.

Reviewer DjDq, who reviewed the previous version, did not submit a review, but stated that they are happy to recommend the paper for acceptance. The editors-in-chief and I agreed that this recommendation is sufficient, so no further review was requested.

Since all reviewers recommend acceptance without further concerns, and since the paper has already undergone multiple revisions (including a major one), I'm happy to recommend acceptance as is.
Decision: Accept (poster)