Perfect density models cannot guarantee anomaly detection
Authors: Charline Le Lan
Laurent Dinh
Venue:  (2021)
Keywords: anomaly detection
out-of-distribution detection
OOD detection
outlier detection
density estimation

Abstract: Thanks to the tractability of their likelihood, some deep generative models show promise for seemingly straightforward but important applications like anomaly detection, uncertainty estimation, and active learning. However, the likelihood values empirically attributed to anomalies conflict with the expectations these proposed applications suggest. In this paper, we take a closer look at the behavior of distribution densities and show that these quantities carry less meaningful information than previously thought, beyond estimation issues or the curse of dimensionality. We conclude that the use of these likelihoods for out-of-distribution detection relies on strong and implicit hypotheses and highlight the necessity of explicitly formulating these assumptions for reliable anomaly detection.

Under review as a conference paper at ICLR 2021
PERFECT DENSITY MODELS CANNOT GUARANTEE ANOMALY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Thanks to the tractability of their likelihood, some deep generative models show promise for seemingly straightforward but important applications like anomaly detection, uncertainty estimation, and active learning. However, the likelihood values empirically attributed to anomalies conflict with the expectations these proposed applications suggest. In this paper, we take closer look at the behavior of distribution densities and show that these quantities carry less meaningful information than previously thought, beyond estimation issues or the curse of dimensionality. We conclude that the use of these likelihoods for out-of-distribution detection relies on strong and implicit hypotheses, and highlight the necessity of explicitly formulating these assumptions for reliable anomaly detection.
INTRODUCTION
Several machine learning methods aim at extrapolating a behavior observed on training data in order to produce predictions on new observations. But every so often, such extrapolation can result in wrong outputs, especially on points that we would consider infrequent with respect to the training
distribution. Faced with unusual situations, whether adversarial (Szegedy et al.) 2013 Carlini & Dietterich 2019), a desirable behavior from a machine learning system would be Hlag these outliers that the user assess if the result is reliable and gather more information need be (Zhao & Tresp 2019 Fu et al.) 2017). This can be critical for applications like medical decision making (Lee et al. 2018) autonomous vehicle navigation (Filos et al. 2020), where such outliers are ubiquitous.
Wagner 2017) or just rare (Hendrycks
What are the situations that are deemed unusual? Defining these anomalies (Hodge & Austin 2004 Pimentel et al.) (014) manually can be laborious if not impossible, and so generally applicable, automated methods are preferable. In that regard, the framework of probabilistic reasoning has been an appealing formalism because a natural candidate for outliers are situations that are improbable or out-of- distriiiitiiiiiionn. Since the true probability distribution density p'x of the data is often not provided,
one would instead use an estimator, p from this data to assess the regularity of point.
Density estimation has been a particularly challenging task on nigh-dimensional problems. However, recent advances in deep probabilistic models, including variational auto-encoders (Kingma & Welling 2014 Rezende et al., 22044, Vahdat & Kautz 2020), deep autoregressive models (Uria et al. 2014 van den Oord et al. 2016b a), and flow-based generative models (Dinh et al. 2014]] 2016 Kingma & Dhariwal 2018). have shown promise for density estimation, which has the potential to enable accurate dessed methods (Bishop 1994) for anomaly detection.
Yet, several works have observed that a significant gap persists between the potential of density-based anomaly detection and empirical results. For instance, Choi et al. (2018). Nalisnick et al. (2018). and Hendryyck et al. (2018) noticed that generative models trained on a benchmark dataset (e.g.,
CIFAR-10, CIFAR-10,Krizhevsky
al. 2009) and tested on another (e.g.. SVHN, Netzer N al. (011) are not
able to identify the latter as -uttt--diitribution with current methods. Different hypotheses have been formulated to explain that discrepancy, ranging from the curse of dimensionality (Nalisnick)
et al. 2019) to a significant mismatch between pX and pX Choi et al. 2018 Fetaya et al. 2020
Kirichenko et al. 2020 Zhang et al. 2020).
In this work, we propose a new perspective on this diserepancy and challenge the expectation that density estimation should enable anomaly detection. We show that the aforementioned discrepancy

---

Under review as a conference paper at ICLR 2021
Figure 1: There is infinite number of ways to partition distribution in two subsets, X and Nout such that Px (Ain) = 0.95. Here, we show several choices for a standard Gaussian pi = (O, 1).
persists with perfect density models. and therefore goes beyond issues of estimation, approxima- tion, or optimization errors (Bottou & Bousquet (008). We highlight that this issue is pervasive as it occurs in low-dimensional settings and for a variety of density-based methods for anomaly detection.
DENSITY-BASED ANOMALY DETECTION
2.1 UNSUPERVISED ANOMALY DETECTION: PROBLEM STATEMENT
1993 Schölkopf et al. 2001), where one aims at distinguishing between regular points (inliers) and irregular points (outliers). However, as opposed to the usual classification task, labels distinguishing inliers and outliers are not provided for training, if outliers are even provided at all. Given a input space X' C RDP, the task can be summarized as partitioning this space between the subset of outliers Nout and the subset of inliers Ne, i.c., Nout U Xn, X' and Nout n Xn = Ø. When the training data is distributed according to the probability measure Px (with density px I one would usually pick the set of regular points Xt, such that this set contains the majority (but not all) of the mass (e.g., 95%) of this distribution, i.e., Px (Xt..)
Unsupervised anomaly detection is classification problem (oyy et al.
(1) 1). But, for any given a, there exists in theory an infinity of corresponding partitions into X₂x and Nowr (see Figure 1). How are these partitions defined to match our intuition of inliers and outliers? We will focus this paper on recently used methods based probability density.
2.2 DENSITY SCORING
When talking about outliers, infrequent observations, the association with probability can be quite intuitive. For instance, would expect anomaly to happen rarely and be unlikely. Since the
language of statistics often associate the term likelihood with quantities like px (x). one might consider unlikely sample to have low "likelihood", that is a low probability density px Conversely, regular samples would have high density px (x) following that reasoning. This is an intuition that is not only prevalent in several modern anomaly detection methods (Bishop) 1994 Blei
ct al. 2017 Hendrycks et al. 2018; Kirichenko et al. 2020 Rudolph et al. 2020 Liu et al. 2020
but also techniques like low-tepperature sampling (Graes 2013) used for example in Kingma
Dhariwal 2018) and Parmar et al. (2018).
The associated approach, described in Bishop (1994), consists in defining the inliers as the points whose density exceed a certain threshold X (for example. chosen such that inliers include a predefined amount of mass, e-g., 95%), making the modes the most regular points in this setting. Nout and Xmn are then respectively the lower-level and upper-level sets (x X, P' (z) & X} and {i E X, pxx (x) X} (see Figure 2b).
WWe will also assume in the of the paper that for any (x) 0.
2

---

Under review as a conference paper at ICLR 2021
P (z)
Hip
A
(a) example of distribution density px.
(b) Density scoring method ap- plied to the distribution px.
(c) Typicality test method (with one sample) applied to the distri- bution px.
Figure 2: Illustration of different density-based methods applied to a particular one-dimensional distribution PX- Outliers are in red and inliers are in blue. The thresholds are picked so that inliers include 95% of the mass, In Figure 2b inliers are considered as the points with density above the threshold À > 0 while in Figure 2c they are the points whose log-density are in the e-interval around the negentropy- (px)
2.3 TYPICALITY TEST
The Gaussian Annulus theorem (Bum et al. 2016 (generalized in Vershynin 2019) attests that most of the mass of a high-dimensional standard Gaussian (0, ID) is located close to the hypersphere of radius vD. However, the mode of its density is at the center O. A natural conclusion that the curse of dimensionality creates discrepancy between the density upper-level sets and what we expect as inliers (Choi et al. 2018 Nalisnick et al. 2019 Morningstar et al. 2020 Dieleman (020)). This motivated
al. (2019) to propose another method for testing whether point an inlier or not, relying on measure of its typicality. This method relies on the notion of typical set (Cover 1999) defined by taking as inliers points whose average log- deg-iiysty is close to the average log-density of the distribution (see Figure 2c).
Definition (Coer 1999.. Given independent and identically distributed elements (x(n) ASN from a distribution with density px. the rypical set ) XNN made of all sequences that satisfy:
H((x)+ N
Σ logp' (2((r)
See
where H(X) -E(logr' (X)) is the (differential) entropy and constant.
This method matches the intuition behind the Gaussian Annulus theorem on the set of inliers of high-dimensional standard Gaussian. Indeed, using a concentration inequality, we can show that
which means that with N large enough, A! (px) will contain most of the mass of (px) ) justifying the name typicality.
- (Pix)igggN (A!N)
(N))
THE ROLE OF REPARAMETRIZATION
Given the anomaly detection problem formulation Subsection 2.1 about the properties solution ought to satisfy, in the ideal case of infinite data and capacity. For density-based methods this means that ( P This setting is appealing as it gives space for theoretical results without worrying about the underfitting overfitting issues mentioned by
we are interested reasoning
Hendrycks et al. (2018); |etaya et al. (2020); Morningstar et al. 2020); Kirichenko et al. (2020); Zhang al. (2020).
Although we work in practice on points (e.g., vectors), is important to keep in mind that these points actually representations of underlying outcome. As random variable, X is by definition the function from this outcome w to the corresponding observation x = (w). However, at its core, an anomaly detection solution aims at classifying outcomes through these measurements. How is the
3

---

Under review as a conference paper at ICLR 2021
Pr(x) Prcx) (f(x))
13
12-
f
T
(a) An example of distribution density Px-
(b) Example of invertible func- tion from [0, 1] to [0.
(c) Resulting density Pf(x) from applying f to X ~ Px as fune- tion of the new axis ((x).
Figure 3: Illustration of the change of variables formula and how much the application of a bijection affect the density of the points considered in doc-eimensionll case. In (igiures 3a and 3c] points x with high density px (x) are blue and points with low density px (I) are in red.
choice of affecting the problem of anomaly detection? While several papers studied the effects of
change of representation through the lens of inductive bias (Kirichenko al. 2020 Zhang et al.) 2020), we investigate the more fundamental effects of reparametrizations f. To sidestep concerns about loss of information (Winkens et al. 2020), study the particular case of invertible map
The measurements x X(w) and f(x = (f o (w) represent the same outcome w (although differently), and, since and f(x) are connected by invertible transformation f. the same method applied respectively to or f(X) should classify them with the same label, either as an inlier or an outlier. The target of these methods is essentially assess the regularity of the outcome From this, we could ideally make the following requirement for a solution to anomaly detection.
Principle. In infinite data and capacity setting, the result of anomaly detection method should be invariant to any continuous invertible reparametrization f.
Do density-based methods follow this principle? To answer that question, we look into how density behaves under a reversible change of representation. In particular, the change variables formula (Kaplan 1952) (used in Tabak & Turner 2013 Dinh al. 2014: Rezende & Mohamed 2015). formalizes a simple intuition of this behavior: where points are brought closer together the density increases whereas this density decreases when points spread apart. The formula itself is written as:
0f d (x)
PTx;; (f(=)) (x)
where af Be. (x) is the Jacobian determinant of f quantity that reflects local change in volume
incurred by f- Figure 3 already illustrates how the function f (Figure 3b) can spread apart points close to the extremities decrease the corresponding density round and I, and, result, turns the density on the left (Figure 3a) into the density on the right (Figure 3c). With this example, one can wonder to which degree an invertible change of representation can affect the density and the anomaly detection methods presented in [Subsections 2.2 and 2.3 that use it.
LEVERAGING THE CHANGE OF VARIABLES FORMULA
4.1
UNIFORMIZATION
We start by showing that unambiguously defining outliers and inliers with any density-based approach becomes impossible when considering a particular type of invertible reparametrization of the problem, irrespective of dimensionality.
Under weak assumptions, one can map any distribution to uniform distribution using an invertible transformation (Hyvärinen & Pajunen! 1999). This is in fact common strategy for sampling from
4

---

Under review as a conference paper at ICLR 2021
PCDE (x) (CDF (x))
12
CDF23
CDF (4)
(a) example of a distribution density px-- Points with high density (x) are in blue and points with low density Px (2) are red.
(b) The corresponding cumulative distribution function F
(c) The resulting density from ap- plying CDPpx to X ~ px is PeDP, px (X) U((0, 1]), there- fore we color all the points the samc.
Figure 4: Illustration of the one-dimensional case version of a Knothe Roohe--Rosenlltt rearrangement, which is just the application of the cumulative distribution function CDFps px the variable
complicated one-dimensional distributions (Devroye 1986). Figure shsws an example of this where bimodal distribution (Figure 4a) is pushed through invertible map (Figure 4b) obtain uniform distribution (Figure 4cp.
To construct this invertible uniformization function, rely on the notion of Knothe-Rosenblat rearrangement (Rosenblatt) [1522 Knothe et al., 1957). A Knothe-Rosenblltt rearrangement (notably used Hyvärinen Pajunen| 1999) is defined for random variable X distributed according to strictly positive density p'x with convex support X' continuous invertible map ((RR) from X onto [0, 1I such that (( (X) follows uniform distribution in this hypercube. This rearrangement (ara T<aa) where CDF, is constructed as follows: Vdl (1, D), (KK (1:) CDFpk 2
the cumulative distribution function corresponding to the density
In these new coordinates, neither the density scoring method nor the typicality test approach can discriminate between inliers and outliers in this uniform D-dimensional hypercube [o, 1] Since the resulting density Pj P)(xr(() is constant, the density scoring method attributes the same regularity every point. Moreover, typicality test on f(KB) (X) will always succeed as
Ve > C.NNNNNN (2(()
H (p))x)(())
1
log logp)(XX)(X) (glKR) (s)
"SN"
N
H (u ( (00 (P.
log(1) 1) =5E.
However, these uniformly distributed points are merely a different representation of the same initial points. Therefore, if the identity of the outliers ambiguous in this uniform distribution, then anomaly detection in general should be as difficult.
4.2 ARBITRARY SCORING
While particular parametrization can prevent density-based -base outlier detection methods from separat- ing between outliers and inliers, we find that it is also possible to build reparametrization of the problem to impose to each point an arbitrary density level the new representation. idea, consider some points from distribution whose density is depicted in Figure 5a and a score function indicated red in Figure 5b) In this example, high-density regions correspond to areas with low score value (and vice- versa). We show that there exists reparametrization (depicted in Figure 5c) such that the density in this new representation (Figure 5d) now matches the desired score, which can be designed mislead density-based methods into wrong classification of anomalies.
illustrate this
5

---

Under review as a conference paper at ICLR 2021
(x))) (o))
(1)
(a) An example of a dis- tribution density px-
(b) The distribution px (in black) and the de- sired density scoring (in red).
(c) A continuous invertible reparametriza- tion f(+) such that f(a X px (x))(x((( s(x).
(d) Resulting density P(()x) from applying
function of (x)).
Figure 5: Illustration of how we can modify the space with an invertible function so that each point x follows predefined score. In Figures 5a 5d points with high density p' (I)) are in blue and points with low density p' (2) are in red.
Proposition 1. For any variable X p' with pix continuous strictly positive (with X' convex) and any measurable continuous function : X + RR, bounded below by a strictly positive number, there exists continuous bijection f(5) such that for any x E X. ((oo()) (x)) everywhere.
(f = s(x) almost
Proof. We write denote (I1,..., (.D.:..,D-11:2) and (I<D, for (II...... (II.....I--.:)) Let f(x X + C R be a function such that
px (x<< dl,
(+))
s s((IeD))))
and Vd (1, (x)) = dd- As s is bounded below, f(+) is well defined and invertible.
By the change of variables formula.
Of(s) Oxt (x)
Px (z)
Yz E X, Prunxy (f( (x(
(x)
s(x).
s(x)
If Xin and Noat are respectively the sets of inliers and outliers, we can pick ball C Xi such that Px (A)
< 0.5, we can choose such that for any x E (X' A), 1 and for any E A, s(x) = 0.1. With this choice of s (or smooth approximation) and the function (() defined earlier, both the density scoring and the (one-sample) typical set methods will consider the set of inliers to be (X') \ A) while Yout (X') \ A), making their results completely wrong. While we can also reparametrize the problem so that these methods may succeed, such reparametrization requires knowledge of (#)) Without any constraints the space considered, individual densities can be arbitrarily manipulated, which reveals how little these quantities say about the underlying outcome
in general.
4.3 CANONICAT. DISTRIBUTION
Since our analysis in Subsections 4. and reveals that densities or low typicality regions are not sufficient conditions an observation to an anomaly, whatever its distribution its dimension, we are now interested in investigating whether additional realistic assumptions can lead to some guarantees for anomaly detection. Motivated by several representation learning algorithms which attempt to learn mapping to predefined distribution (e.g.. standard Gaussian, see Chen & Gopinath| 2001 Kingma & Welling 2014 Rezende et al. 2014 Dinh et al., 2014 Krusinga et al. 2019) we consider the more restricted setting of fixed distribution of choice, whose regular regions could for instance be known. Surprisingly, we find that it is possible to exchange the densities of inlier and outlier even within canonical distribution.

---

Under review as a conference paper at ICLR 2021
Xe
Nout
(a) Points and in uniformly distributed subset. p(rat) will pick a two-dimensional plane and use the polar coordinate using the mean Zmm of Zin and Vout as the center.
Applying bijection f(ret) exchanging the points and Zout. f(rot) is rotation depending the distance from the mean Zm. of Zin and Sout in the previously selected woodiimensional plane.
Figure 6: Illustration of the norm norm-dependent rotation, a locally-acting bijection that allows to swap two different points while preserving uniform distribution (as a volume oolum--preserving function).
Proposition 2. For any strictly positive density function P'x over a convex space N' C. IRD with D 2, for any Zins Yout the interior X of X, there exists a continuous bijection such that x'x Pyyy P())) (f (x((n)) px (zlout)), and ())) (z)tott
X
(z!out))) p' (z((in).
We provide sketch of proof and put the details Appendix A We rely on the transformation depicted in Figure 6) which can swap two points while acting in very local area. If the distribution of points is uniform inside this local area, then this distribution will be unaffected by this transformation. In order to arrive at this situation, we use the uniformization method presented in Subsection 4.1] along with a linear function to fit this local area inside the support of the distribution (see Figure 7}. Once those two points have been swapped, we can reverse the functions preceding this swap to recover the original distribution overall.
Since the resulting distribution p)) is identical the original fxx then their entropies the same H (P)x)) H (. Hence, when Vin and Tout are respectively an inlier and an outlier, whether terms of density scoring or typicality, there exists a reparametrization of the problem conserving the overall distribution while still exchanging their status as inlier/outlier, We provide an example applied to a standard Gaussian distribution in Figure
L(Xouz) L(xm))
Xout
xm
L(xi)
Xin
(a) When taking points Zio and Vout inside the hypereube [0, 1] there is sometimes no Lalball cen- tered in their mean on containing both Zin and Tout.
(b) However, given Tox and Fout, apply in- vertible linear transformation such that there exists La-ball centered in their new mean L(xm) contain- ing both L(xx,)) and L(Xout), If the distribution was uniform inside [o, 1) then is now also uniform inside L ((0, 1)
Figure 7: We illustrate how, given and Yout in uniformly distributed hypercube [0, 1) one can modify the space such that f(rot) shown in Figure 6 can be applied without modifying the distribution.

---

Under review as a conference paper at ICLR 2021
(f(z))
(f(z))
(f(z)),
(f(z))
(a) Points sampled from pk (0,, l2).
(b) Applying bijection f that pre-(e) The original distribution pi with serves the distribution Pi()) respect to the new coordinates f(),
(0, I2) the points in Figure 8a
px f-1
Figure 8: Application of a transformation using the bijection in Figure 6 to standard Gaussian distribution N I2), leaving it overall invariant.
This result is important from a representation learning perspective and a complement to the general non-identifiability result several representation learning approaches (Hyvärinen Pajunen 1999; Locatello et al. 2019). It means that learning representation with a predefined, well-known distribution and knowing the true density PX are not sufficient conditions to control the individual density of each point and accurately distinguish outliers from inliers.
5 DISCUSSION
Fundamentally, density- methods for anomaly detection rely on the belief that density, quantity, conveys useful information to assess whether an outcome is an outlier or not. For example, several density-based methods operate in practice on features learned independently from the anomaly
detection task (Lee et al., 2018 Krusinga et al. 2019 Morningstar et al. [020 Winkens et al. 2020) or on the original input features (Nalisnick et al. 2018 Hendrycks
Kirichenko al.)
2020, Rudolph et al. 2020 Nalisnick et al. (201)). In general, there is evidence that the density in these representations will earry any useful information for anomaly detection bringing into question whether performance of probabilistic models on this task (e.g.. Du & Mordatch 2019 Grathwohl
(ct al.) 2019 Kirichenko et al.) 20200 Liu & Abbeel 2020) reflects goodness-off ofosss--fffr of the density model. On the contrary, have proven in this paper that density-basee anomaly detection methods are inconsistent across a range of possible representations/ P even under strong constraints on the distribution, which suggests that finding the right input representation for meaningful density-based anomaly detection requires privileged information, as discussed in Subsection 4.2 Moreover, several papers have pointed existing problems in commonly used input representations; for example, the geometry of bitmap representation does not follow our intuition of semantic distance (Theis et al. 2016), or images can come from photographic sensors tuned specific populations (Roth [2099] Buolamwini & Gebru 2018). This shows how strong of an otherwise understated assumption it is to suppose that the methods presented Subseccioo 2.2 and [Subsection 2.3 would work on input representations. This is particularly problematic for applications as critical as autonomous vehicle navigation or medical decision makiin....
While defining anomalies might be impossible without prior knowledge WWnnkens al. 2020 as out-of-distribution detection an ill-posed problem (Choi al., 2018: Nalisnick et al. 2019
Morningstar et al. 2020), several approaches make these assumptions more explicit. For instance, the density scoring method has also been interpreted in Bishop (1994) as a likelihood ratio method (Ren al. 2019 Serrà al. 2020 Schirrmeister al. 2020), which, the one hand, relies heavily the definition of an arbitrary reference density as denominator of this ratio but, on the other hand, is invariant to reparametrization. Inspired by the Bayesian approach from Choi et (2018). one can work on defining a prior distribution on possible reparametrizations over which to average (similarly
Jorgensen Hauberg 2020).
A Iternatively, this can be seen change of base distribution used to define probability density as Radon- -ikiodym- derivative.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings- Vision, Image and Signal processing, 141(4):217-22,, 1994.
David Blei, Katherine Heller, Tim Salimans, Max Welling, and Zoubin Ghahramani. Panel: On the foundations and future of approximate inference. In Symposium on Advances in Approxi- mate Bayesian Inference. AABI 2017, 2017. URL https: www. youtube. com/ ch?
V=x1UByHT WwQ.U wew.urreeaaaaaaeeeeeeeeeee
be&t=4 6m2s
Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science. Vorabversion eines Lehrbuchs, 5, 2016.
Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. Advances neural information processing systems, pp. 161-168, 2008.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on faimess, accountabilityy and transparency, pp- 77-91, 2018.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 3-14, 2017.
Scott Saobing Chen and Ramesh A. Gopinath. Gaussianization. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 423-429. MIT Press, 2001. URL http:
nips. wwtppcpppppppppppcccccc /
zat ion. pdf
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392. 2018.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Lue Devroye. Sample-based non-uniform random variate generation. In Proceedings of the 18th conference on Winter simulation, pp. 260-265. ACM, 1986.
Sander Dieleman. Musings typicality, 2020. URL http:ppp////eeeeeee.c..oooooo/// 09/01/Eypicallty..ttm
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.85.1 2014.
Laurent Dinh, Jascha
and Samy Bengio. Density estimation using real nvp. arXiv
preprint
2016.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, pp. 3608-3618, 2019.
Ethan Fetaya, Joern-Henrik Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the limita- tions of conditional generative models. In International Conference on Learning Representations, 2020.
Angelos Filos, Panagiotis Tigas, Rowan McAllister, Nicholas Rhinchart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? arXiv preprint arXiv:2006. 149/1, 2020.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. In Advances in neural information processing systems, pp. 2577-2587, 2017.
Will Grathwohl, Kuan-Chich Wang. Jöm--Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03226. 2019.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.085: 2013.

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL [RLLiittpp://eeeevveee.. net/ wtt.::/tttooouunttoc-liii211.11.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 0.....2..0666 2018.
Victoria Hodge and Jim Austin. A survey of outlier detection methodologies. Artificial intelligence review, 22(2):85-126, 2004.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-43, 1999.
Martin Jørgensen and Søren Hauberg. Reparametrization invariance in non-parametric causal discovery. arXiv preprint Stt1:2000800555222 2020.
Wilfred Kaplan. Advanced calculus. Pearson Education India, 1952.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR'2014,
arXiv:1312.61 14, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions, In Advances in neural information processing systems, pp. 10215-10224, 2018.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect outtof-disttibuutio data. arXiv preprint arXiv:2006.00 08545, 2020.
Herbert Knothe et al. Contributions to the theory of convex bodies. The Michigan Mathematical Journal, 4(1):39-52, 1957.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ryen Krusinga, Sohil Shah, Matthias Zwicker, Tom Goldstein, and David Jacobs. Understanding the (un) interpretability of natural image distributions using generative models. arXiv preprint arXiv:1901.01999 2019.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pp- 7167-7177, 2018.
Hao Liu and Pieter Abbeel. Hybrid diseriminative-geneeeee training via contrastive learning. arXiv preprint +rXiv:2007.09070, 2020.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based
detection.
Advances in Neural Information Processing Systems (NeurIPS), 2020.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of dis- entangled representations. In international conference on machine learning, pp. 41 4114-41 4124, 2019.
Warren R Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alexander Alemi, and Joshua V Dillon. Density of states estimation for out-of-distribution detection. arXiv preprint Xrvi:22006.0m 09273, 2020.
Mary M Moya, Mark W Koch, and Larry D Hostetler. One-class classifier networks for target recognition applications. STIN, 93:24043, 1993.
Eric Nalisnick. Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv: 1810.09130 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXiv preprint arXiv:1906.02994 2019.
10

---

Under review as a conference paper at ICLR 2021
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. Proceedings of Machine Learning Research, pp. 4055-4064, Stockholmsmässan, Stockholm Sweden, 2018. PMLR.
Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection, Signal Processing, 99:215-249, 2014.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14707-14718, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of Machine Learning Research, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of Machine Learning Research, 2014.
Murray Rosenblatt. Remarks on a multivariate transformation. The annals of mathematical statistics, 23(3):470-472, 1952.
Lorna Roth. Looking at shirley, the ultimate norm: Colour balance, image technologies, and cognitive equity. Canadian Journal of Communication, 34(1), 2009.
Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semi-supervised defect detection with normalizing flows. arXiv preprint arXiv:2008. /2577, 2020.
Robin Tibor Schirmeesster, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. arXiv preprint arXiv:2006. 10848, 2020.
Bernhard Schölkopf, John C Platt, John Shawe-Tayler Alex J Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural computation, 13(7)1144311471, 2001.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distributien detection with likelihood- based generative models. In International Conference on Learning Representations, 2020. URL /
net/ fert/Eorum? 1d=idd=xxxii INpVYvr
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing propertics neural networks. arXiv preprint arXiv:1312.6199, 2013.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. note on the evaluation of generative models. In ICLR' 2016 arXiv:1511.01844 2016.
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, pp. 467-475, 2014.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, al. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, pp. 4790-4798, 2016a.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016b.

---

Under review as a conference paper at ICLR 2021
Roman Vershynin. High-dimensional probability, 2019.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566. 2020.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition, arXiv preprint arXiv:2003. /2506, 2020.
Rui Zhao and Volker Tresp. Curiosity driven experience prioritization via density estimation. arXiv preprint arXiv:1902.0803: 2019.
12

---

Under review as a conference paper at ICLR 2021
A PROOF OF PROPOSITION 2
Proposition For any strictly positive density function pix over a convex space d' C TR. with D 2. for any Vins Vow the interior X of X, there exists a continuous bijection : X'' such that p'x = PJixy P)(x) (f (xlout)), and P)(() (f (z!oa0)) - p'x (z(in)).
=
Proof. Our proof will rely on the following non-rigid rotation f(rot), Working in hyperspherical coordinate system consisting of radial coordinate and (D - 1) angular coordinates (o.)
d-1 sin (oi) cos(óa)
PA < D, Iz = r
D-2
I
sin (i)
Sin(ØD-1),
where for all i E 2.., D 2), di E [0, 1) and Φn E [0, 2T), given Tman > ro > 0, we define the continuous mapping f(rot) as:
(Tman - r)+ mod 2T] Φn 2, Φn- + f((rat) ((r, Φ1 on On-2.ón-11 = r, ,
maaz To
where (+) max 0). This mapping only affects points inside (2, (mon) ). and exchanges two points corresponding to (ro, $1,..., 22 1-) and (ro, .1...., "......pp-.... 2D- 2n-1 + π in continous way (see Figure 6). Since the Jacobian determinant of the hyperspherical coordinates transformation is not a function of ΦD-1, f(rat) is volume preseerrng in cartesian coordinates.
Let (KK be Chothh-RRosenblate rearrangement of pxx
((KR) P(K (X) is uniformly distributed in [o, 1j2
Let (in)) f(K (x)) and 2(aut) = f(Kn) (z(out).. Since f(Kn) is continuous, z((m), (out)
in the interior (0, 1) Therefore, there is
such that the L2-balls Bg (2(im), e) and
B2 (2 (out)) ()) are inside (0, P. Since (0, 1)P convex, so is their convex hull.
Let ro = ||=((m)) - _(oue)|| 2 and Imar = ro + 6. Given 2 E 10, we write 2| and 21 to denote its parallel and orthogonal components with respect (2(1m) 2(out)). We consider the linear bijection L defined by
((=) i|| + 1
Let f(=) = L. f(KR) Since L is linear function (i.e., with constant Jacobian), (X) is uniformly distributed inside L (Io. 1). If 2(() is the mean of 2())) and (m then (X) contains B, (I. (z(m)) .mmax) (see Figure 7h. We can then apply the non-rigid rotation f(rot) defined
earlier, centered on (z(m)) to exchange (z())) and L (z(oa))) while maintaining this uniform distribution.
We can then apply the bijection (f(=)) to obtain the invertible map f= -f(x)) f(rot) o f(x) such that p)(x) P((x) (f (z(in)))
pX (x((t))). and p((()) (f (z(out))) px (z((i)). I
13


Reviews:
[{"review_id": "ykK2i3DJRVm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Quality*\nI have a major concern regarding the principle stated on the page 4 that the whole paper discussion is based on. \nFirst, I totally disagree that anomaly detection methods should be invariant under any invertible reparametrizations. In practice, quite often the very definition of an anomaly detection problem is tighted to a concrete data representation.\nIn addition, anomaly detection can be thought of as a binary classification and the stated principle amounts to saying that a binary classifier should provide the same results for any (invertible) data re-parametrization, e.g. using various feature extractors. This is clearly wrong thing to ask from a classifier and instead people focus on finding useful feature extractors. Hence, in the case of anomaly detection, it comes at no surprise that one can re-parametrize the data so that the anomaly detection algorithm (density or not density based) is totally confused.\n\nWhile the paper has interesting discussions and constructions of the worst case scenarios for confusing density-based anomaly detection methods, I don’t think that the paper uses them to support a useful claim. \n\n*Clarity*\nThe paper is well-written in general. \n\n*Originality*\nTo my best knowledge, the theoretical study of the role of data re-parametrization for density-based anomaly detection is new.\n\n*Significance*\nI don’t think that the results of the paper are significant in their current form and might cause a confusion with its statements.\n\nI would encourage the authors to clarify their view on the relation between anomaly detection and the underlying data representation and clarify/justify the principle from the page 4.\n\n\nPros\n* The paper presents a collection of interesting transformations between densities\n\nCons\n* The main message of the paper is based on unrealistic principle\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "f4yS3CqoZVk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a negative result that anomaly detection by density estimation is flawed.  \n\nAfter highlighting several problems with anomaly detection, the paper focuses on a seemingly plausible invariance argument, that the lack of invariance of a distribution's random variable to change of variable - reparameterization - makes it impossible to define which events fall in dense or rare areas of the distribution. Specifically it proposes the principle that an outcome represented by a random variable X or it's transformation fX should not result in change of it's classification as outlier versus inlier.  As the paper demonstrates, it is not difficult to devise a counter example of a transform f that can flip an outcome between inlier and outlier with weak conditions on any distribution of X. \n\n The paper identifies inliers by introducing \"typicality\" as a measure. Although interesting, it contributes little to the paper's argument and unlike the level set definition of inliers, typicality does give not a constructive tool for anomaly detection. \n\n However is any practical sense, the invariance argument is specious. Consider this example, of the distribution of sunlight over the earth. If one is interested in the effects of sunlight on population health (e.g. cancer rates) one looks at the distribution of intensity over individuals. Alternately if the question is the incidence over area (solar power potential) then its the random variable of land area that is relevant.  There is an invertable deterministic function between land area and population random variables given by the population per area . Clearly a place (think outcome) with low cancer incidence (an outlier) because of sparse population may be a good place for solar power. Hence the parameterization of random variable for anomaly detection is intrinsic to the problem rather than serving as a condition for invariance. \n\n In a related area, invariance as applied to prior distributions has been studied extensively, as found in E. T. Jaynes, \"Probability Theory\" (Cambridge, 2003), ch 12, to show when invariance is applicable. \n\nI have no issue with the paper's rigor and derivations, rather the issue is with the unfounded presumption that invariance to reparameterization inhibits the use of density based anomaly detection. As the paper states in the final discussion, \"defining anomalies might be impossible without prior knowledge\"-- this may not be a bad thing, and the same material used to argue for a knowledge-based approach to anomaly detection that includes specifying a random variable appropriate for the domain would be a valuable, acceptable paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BZc4sdjuA27", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Detecting anomalies is a notoriously ill-defined problem. The notion of anomaly is not a rigorous concept and different algorithms produce different results. The paper critiques a broad set of methods which involve likelihood (or density) estimations. It's main idea revolves around the 'Principle' set on Page 4. The principle claims that when data capacity and computational constraints are removed, an AD algorithm should be invariant to 'reparametrization' of the input. Roughly speaking, that means the algorithm should be invariant to arbitrary 'name changing' of the input - the result should not change if each data item x is replaced by f(x) if f is invertible. \nThe paper then shows that density models do not satisfy this principle even when they are 'perfect'.\n\nMy main critique of the paper is that this principal constitutes a completely unreasonable requirement of any AD algorithm, to the point where it is meaningless. It is trivial to observe (as the authors do) that any continuous distribution could be transformed to a uniform distribution with the correct f. Even if the domain is discrete we can make the input uniform: As an example think of f being a Pseudo-Random function, like say f(x) is a digital signature of x. If we believe cryptography then no efficient algorithm would be able to say anything useful. \n\nTo sum up - I don't think the 'principle' is a useful prism by which to measure models and definitions, and therefore I don't find the contribution of the paper sufficient for publication.\n\nAs a side remark, this principle may be useful if further constraints are put on f, for instance, if we may want the AD algorithm to be oblivious to unit change in the data, which translates to f which multiplies dimension be a constant. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1v3mI-oKtFQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**\n\nMy impression after the extensive discussion is that the remaining differences are possibly too subjective to come to an agreement:\n\n1) Whether the fact that the invertible reparameterization principle does not hold for anomaly detection represents a significant theoretical contribution. To me, I still don’t quite see why, but I do believe there are other researchers with a stronger theory background more qualified than me to evaluate this, I will therefore downgrade my experience score by 1.\n2) Whether the fact that the invertible reparameterization principle does not hold has promise to further understand the practical issues like the CIFAR10 vs SVHN phenomenon beyond things that have already been discussed in the literature - I think quite likely not, the different arguments for and against have been discussed in our back and forth I think.\n\nI invite the AC to go through the discussion here to see the detailed arguments/rebuttals.\n\nMy personal impression, in case the authors are interested, is also the manuscript might be further improved by giving more space to these two questions, i.e. the motivation of the principle and instead moving some of the proofs to the supplementary. I think it is quite straightforward to understand that the principle cannot hold and more difficult to understand why it should hold in the first place.\n\n\n**Summary**\nIn this manuscript, the authors highlight one phenomenon about density-based anomaly detection: Even the perfect density model can assign arbitrary densities if you allow an arbitrary invertible reparametrization of your data before estimating the densities and therefore will have arbitrary inlier/outlier classification. Therefore, methods that assign anomalies purely based on low density values cannot achieve a principle proposed by the authors: “anomaly detection methods should be invariant to any invertible reparametrization f”.\n\n**Main Impression**\nThe manuscript is well-structured and well-written, allowing me to quickly understand the main argument of the paper (I hope), thanks for that. I found the analyses to be technically correct to the level I was able to analyze them.\n\nOverall, I have some doubts about how much new practically relevant things can be learned about anomaly detection from the argument laid out in the paper.\n\n**Details about Main Criticism**\n1) The fact that densities can be arbitrarily changed by invertible functions is already an important building block of normalizing flows  - in order to learn any distribution starting from a predefined prior like a gaussian distribution. So in itself it does not seem like a new aspect to me. \n2) The effect of invertible reparametrizations on anomaly detection has already surfaced in the literature: E.g., in Nalisnick, 2018, they already report very different anomaly detection results if one uses only p(z) and ignores the log-determinant for an affine Glow Network. This setup there, as far as I see, fits in the framework here, with f there implemented by the affine Glow network. \n3) I am not sure I agree with the principle that anomaly detection methods should be invariant to invertible reparametrizations. An example: Let’s say you visualize the samples from a gaussian distribution as grayscale values, with gray at zero, and positive values becoming brighter, negative values becoming darker. Then very dark values as well as very bright values would be visually outliers. Now imagine you do an invertible reparametrization such that the distribution becomes bimodal with the modes further away from the center. Visually, originally somewhat darker/brighter values would be stretched to contain a lot of very dark/very bright values. Now some very dark values/very bright values would no longer be outliers in the perfect density model, but I would argue they would also visually no longer be outliers. \n4) Further, the original parametrization of e.g., the way images are stored is not arbitrary, but a parametrization where we as humans can perceive the content. \n5) Note also that in e.g., normalizing flows literature, it is always attempted that the estimated density is a lower bound for the discrete probabilities. There are also models directly modelling the discrete probabilities, e.g., see https://arxiv.org/abs/1905.07376 https://arxiv.org/abs/2006.12459. Note that normally, for density-based models one can compare the continuous and discrete models, while the proposed reparametrization logic here is not applicable for discrete models at all. \n6) The logic also rests on the assumption of the “true” density always being not zero/positive (footnote page 2). For the phenomena the paper references, I am not sure this is a good assumption. E.g., should SVHN-like zeros really have positive density under the true density of CIFAR-like frogs? One could also argue they have different support?\n7) In my view, for the case of images, the anomaly detection phenomena are already well-explained by the line of thoughts in Ren et al., 2019; Serrà et al., 2020; Schirrmeister et al., 2020 and Kirichenko et al., 2020 so I am not sure in which scenarios the invertible-reparametrization aspect adds further understanding.\n\nTo summarize again, my main problem is, I am not convinced that “the result of an anomaly detection method should be invariant to any invertible reparametrization f” and that the failure to do so by density-based methods helps understand their failures as pointed out in recent literature.\n \n**Suggestion for Improvement**\nWhat would change my mind about the value of the phenomenon highlighted here is if there are examples in the literature where people use densities that are strongly affected by a reparametrization and are not aware of it. I think such examples would also be more useful rather than showing it also works with canonical distribution etc. I think the entire principle with invertible reparameterizations is clear enough, and any analyses showing it works in many mathematical settings, could be moved to supplementary. The more important question to me is when this principle makes sense at all.\n\n\n**Other Points**\nIs the whole setting in 2.1 one also used in literature? References might be helpful for “one would usually pick the set of regular points X_in such that this set contains the majority (but not all) of the mass (e.g.,95%) of this distribution”\n\n“Without any constraints on the space considered, individual densities can be arbitrarily manipulated, which reveals how little these quantities mean in general.”\n-> I don’t agree this shows how little densities mean in general. I think it shows how little they mean once you allow arbitrary invertible reparametrizations, and that maybe allowing this is not a good idea ;)\n\n“It means that learning a representation with a predefined,  well-known distribution and knowing the true density p∗(X) are not sufficient conditions to control the individual density of each point and accurately distinguish outliers from inliers.”\n-> I think this statement is a bit strange. If you know the true density at original parametrization and have access to the data in original parameterization, you can evaluate it and accurately distinguish outlier/inlier if they are defined by density-levels there in the first place. Also if they are invertibly transformed and you know the invertible transformation you can also invert it… if they are not defined by density-level at original parametrization, then the question is what are they defined by anyways, and you have a whole new question in my view.\n\nI encourage the authors to also clear up any misunderstandings of the current manuscript from my side.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Firstly, thank you authors for your thought-provoking submission and discussion.  The key point of disagreement clearly is the fundamental assumption that "the result of an anomaly detection method should be invariant to any continuous invertible reparametrization f."  All reviewers found this assumption to be too strong, leading all four to recommend rejection.

I also recommend rejection at this time.  To me, it seems reasonable and practical to assume that anomalies are defined based on distance (in a fixed feature space).  So if we are allowed to deform the space, clearly this definition breaks down and the concept of an anomaly becomes empty.  Perhaps I am wrong about this, but nevertheless, the paper could do a much better job of convincing the reader that its fundamental reparametrization assumption is appropriate and of consequence in practice.
Decision: Reject