The Tilted Variational Autoencoder: Improving Out-of-Distribution Detection
Authors: Griffin Floto
Stefan Kremer
Mihai Nica
Venue: ICLR 2023 poster (2023)
Keywords: Variational Autoencoder
OOD
Unsupervised

Abstract: A problem with using the Gaussian distribution as a prior for the variational autoencoder (VAE) is that the set on which Gaussians have high probability density is small as the latent dimension increases. This is an issue because VAEs try to attain both a high likelihood with respect to a prior distribution and at the same time, separation between points for better reconstruction. Therefore, a small volume in the high-density region of the prior is problematic because it restricts the separation of latent points.  To ameliorate this, we propose a simple generalization of the Gaussian distribution, called the tilted Gaussian, which has a maximum probability density occurring on a sphere instead of a single point. The tilted Gaussian has exponentially more volume in high-density regions than the standard Gaussian as a function of the distribution dimension. We empirically demonstrate that this simple change in the prior distribution improves VAE performance on the task of detecting unsupervised out-of-distribution (OOD) samples. We also introduce a new OOD testing procedure, called the Will-It-Move test, where the tilted Gaussian achieves remarkable OOD performance.

Published as a conference paper at ICLR 2023
THE TILTED VARIATIONAL AUTOENCODER: IMPROVING OUT-OF-DISTRIBUTION DETECTION
Griffin Floto
Stefan Kremer, Mihai Nica University of Guelph (sk (skremer, ni nicam @uogue 1pheepphc.cc
University of Toronto
gri triffin. floto@mal
i 1, utoronto.ca
ABSTRACT
A problem with using the Gaussian distribution prior for variational autoen- coder (VAE) is that the set on which Gaussians have high probability density is small the latent dimension increases. This is an issue because VAEs aim achieve both a high likelihood with respect to prior distribution and at the same time, separation between points for better reconstruction. Therefore, small vol- ume in the high-density region of the prior is problematic because it restricts the separation of latent points. To address this, we propose simple generalization of the Gaussian distribution, the tilted Gaussian, whose maximum probability density occurs on a sphere instead of single point. The tilted Gaussian has exponentially more volume in high-density regions than the standard Gaussian as a function of the distribution dimension. We empirically demonstrate that this simple change in the prior distribution improves VAE performance on the task of detecting un- supervised ouist--distriuutio (OOD) samples. We also introduce new OOD testing procedure, called the Will-It-Move test, where the tilted Gaussian achieves remarkable OOD performance.
INTRODUCTION
Due to its simplicity, the Gaussian distribution is a common prior for the variational autoencoder (VAE) (Kingma & Welling 2014 Rezende et al., 2014). One drawback it has is that the region of high probability density becomes relatively smaller why this is an issue, consider the objective of the VAE. It tries to encode points such that they are close to the prior and can reconstruct points into their original form. Given a limited capacity of an encoder/decoder model, points in the latent space must be separated to have significant differences in their reconstructed points. With a sufficiently complex data set, it would be required to have a large volume in the high density region of the prior distribution to accommodate each of the latent points, while allowing for sufficient separation.
the latent dimension increases. To see
We argue that the Gaussian distribution's 's volume under regions of high probability density is not large enough to accommodate real data sets. To this end, we show that many of the points encoded by Gaussian prior VAEs exist in low-density regions, and that the high-density region remains rela- tively empty. In support, Nalisnick et al. (2019a) report that the latent point at the highest density of Gaussian VAE trained on MINST was an all-black image. To deal with these issues, we propose a simple generalization of the Gaussian distribution called the tilted Gaussian distribution. We cre- ate this distribution by "exponentially tilting" the ordinary multivariate Gaussian distribution by its norm. The operation of exponential tilting is common procedure in such diverse fields as statistical mechanics, large deviations or importance sampling, but we believe using for VAES as we do here is a novel contribution. The tilted Gaussian has maximum probability density lying on the surface of a sphere rather than at a single point. A single parameter corresponds to the sphere's 's radius, allowing for control of the volume under the high-density region of the distribution. We show that the tilted Gaussian has exponentially more volume that the standard Gaussian as as function of the latent dimension, allowing for far greater proportion of points from dataset to exist in regions of

---

Published as a conference paper at ICLR 2023
Gaussian VAE
Tilted VAE
4
30
42
8
2
10
R25h
Ro0%
RyS%
-20
-2
-30
-20 -10
10 30
Figure I: 2D representation of the 10D latent space of the Gaussian VAE vs the tilted VAE with 25 trained on the Fashion-MNIST dataset, plotted with isoradial projection that preserves the radius (i.c. T2D = || ||; see Appendix [.1 for details on this isoradial projection). Shaded regions indicate where the latent probability density is at least 25%, 50% or 75% respectively of its maximum, R. == {= p(z) 2 cmaxw p(w)). The encoded points of the tilted VAE lie almost entirely in the region of high probability density, while the ordinary Gaussian VAE places points
outside these regions. See also Section 3.5 for more comparisons.
high probability density. We investigate a simpler method of increasing the prior volume by using Gaussian with large variance, however the effect on performance was minimal D.2
To demonstrate the benefits of the tilted Gaussian as a prior for the VAF, we focus on the task of OOI) detection. It has been noted that somewhat surprisingly, VAFs assign high likelihood OOD points, despite being optimized
lower bound of the log- log-likelihood (Nalisnick et al. 2019a Choi et al. 2019). A possible contributing factor to this poor performance is that the high-density region of the latent space is not densely populated by in-distribution (ID) points due to the volume considerations previously detailed. We show that VAEs using the tilted Gaussian as prior (which call the tilted VAE), have a far greater percentage of points in high density regions (see Figure for an illustration and Section 3.5 detailed numbers) and perform significantly better on the OOD task (See Table 2 While the improvement is step towards robust OOD detection with VAEs, show that a prior distribution alone cannot achieve the desired level of performance on this task. Thus, we propose new lest, called the WilveIt-Move test, for the OOD problem. Combined with the tilted Gaussian as prior, it consistently improves the performance of current methods to perform OOD detection with VAEs. (See Section 4.2 for description and Table 2 for results)
2 RELATED WORK
2.1 EXTENSIONS OF VARIATIONAL AUTOENCODERS
In this section, we give non-exhaustive list of extensions proposed for VAEs. The majority of approaches aim to increase the flexibility of the prior. For example, a mixture of Gaussians can be used as alternative (Dilokthanakul et al.) 2016) to the standard Gaussian prior. The VampPrior attempts to improve upon the mixture Gaussians by using a mixture of variational posteriors Tomczak & Welling 2018). Another proposal for the prior distribution is the hyperspherical VAE Davidson et al. 2018). which uses a Con-Mises-Fiiher (VMF) distribution. In contrast, our tilted prior concentrated around the hypersphere as "soft" constraint, which allows the ordinary normal distribution instead of the VMF to be used. Other proposed methods use the Dirichlet process Nalisnick & Smyth 2017). the Chinese restaurant process (Goyal al. 2017). and the Gaussian
2

---

Published as a conference paper at ICLR 2023
process (Casale et al., 2018). Hierarchical constructions of priors can be used to construct prior distributions as in Sønderby et al. (0166), Maalée et al. (2016), and Maalóe et al. (2019), using various methods construct the latent variable hierarchy. A different approach to enforce lower bound on the Kullback-Leibler divergence (KI.D) term of the VAE, taken by the delta- VAE (Razavi et al. 2019). A powerful way to achieve greater flexibility is to normalizing flows, class of invertible transformations that can be used to construct more complex posteriors.
2.2
SETTOF--DITTRBBUTION DETECTION METHODS FOR VARIATIONAL AUTOENCODERS
Arguably the simplest approach to perform OOD detection with VAËS is by using a one-sided test based on the likelihood that VAFs assign to data points (Bishop 1994). Given that this method per- forms surprisingly poorly on the OOD task Nalisnick al. 2019a), a number of alternative scores have been provided. Likelihood Ratios (Ren et al. 2019) proposes to use the ratio between two dif- ferent types of models, one capturing the semantic content of data, the other capturing background information. Likelihood Regret (Xiao et all.2222) uses a similar principle, but uses the ratio between an model that is optimized for the training dataset and another optimized for an individual sample. ROSE (Choi et al.) 2021] uses a method to compute how much sample would update model's s parameters. Input complexity (Serrà et al. 2020) use an estimate of the Kolmogorov complexity as well as the log- likeliiold estimate assigned by the VAE. Density of States Estimation (Morningstari
(ct al.] (021)) uses an approach based on statistical physics and directly measures the typicality of different model statistics to classify OOD samples. Nalisnick al. (20T9b) uses typicality test and Song et al. (2019) uses batch normalization statistics. |Ran et al. (2021) takes a different approach by adding Gaussian noise images and employs noise contrastive prior the VAE architecture.
3 THE TILTED GAUSSIAN DISTRIBUTION AND THE TILTED VAE
3.1 REVIEW OF THE VARIATIONAL BOUND USED FOR TRAINING VAES
Consider a dataset X = [x') consisting of N i.ddd samples of some distribution in Rd=, We
model the data by two-step process: first, an unobserved latent variable E Rdd is sampled according to a prior distribution pz (=); second, a value x'' is produced conditionally on the latent by parameterized generator ("decoder") pe (xr according some unknown parameter 0 0"
To attempt to recover 0*, we maximize the marginal fog-likelihook of the data E log Pe (x") =
Σ Sae, log (og"((=='=)p> (=)dz. As the integral over the latent variables is often intractable, the variational lower bound is used to introduce a parameterized inference model ("encoder") qs (z|xx to approximate the true posterior po (z(xr). Given an encoder model qa- the variational lower bound of the log- legllikelihood! is
N
N
Σ log pe (x") 2 Σ (xola)))) [log pe (z/1z)] - Σ DKL Pz(2)),
(I)
1=1
where DKL. (.) is the Kullback-Leibler divergence (KLD). The KLD term in the equation above can be interpreted as fitting the aggregated posterior 2s (=la*) to the pre-determined prior pz (=)- Typically, the standard Gaussian distribution (z(2)= ~ ~ (0, f) is used as the prior and the encoder distribution is qo (2|z) ~ - (μ(z), diag(o?(() (z))) with parameterized functions u(xx), <²(x) E Rd for the encoder mean and variance. In this case, the KLD term appearing in (T) can be evaluated explicitly as
1 de
1 DRL (H, diag(o?)) (I (
-111111 + Σ 2
(of - 1ogo
3-1
this formula makes the lower bound of 1 explicit, and be optimized using stochastic gradient descent. Gaussian noise is added to each point during training so that the latent point 2¹ assigned to the data point x' is z ((x')) + (N 7), see e.g. (Kingma & Welling 2014).
3.2 IMPROVING THE VAE
The primary technical contribution of this paper is the proposal of an alternative prior distribution pz (z), the exponentially tilted Gaussian prior. As the standard Gaussian prior has maximum density
3

---

Published as a conference paper at ICLR 2023
at the origin, the KLD term in the log-likelihood bound (2 forces all latent points into the same location, 14 = 0 and o, = 1. This leads to "crowding" around the origin, making it difficult to differentiate between encoded data samples (Hoffman & Johnson, 2016 Alemi et al. 2018).
Unlike the standard Gaussian, the tilted prior does not have maximum density at a single point. Instead, the maximum density occurs at all the points on the hyper-sphere of radius illustrated in Figure 2 This allows the model to spread out latent points while still optimizing the marginal log-likelihood bound (I.
Additionally, the radii ||=||| of points drawn from this distribution are near 7 with high probability TT
Therefore, the norm of encoded point ||=|| drawn from (0(2(2r) can form simple statistic which is an effective test for OOD points (see (7)). We also obtain new variational bound. the analogue of equation 2 for the KLD of the tilted Gaussian distribution. This allows training the tilted VAE by simple modification to the standard Gaussian. This new prior can substitute for the standard Gaussian with minimal changes existing code. In practice, the only change replacing the term 2 |||ll in the KLD from equation 2 to (I|a] || II)² where ||~*| is fixed constant
depending the tilting parameter т. This corresponds to the KLD of the new tilted model in equation (6).
3.3 EXPONENTIALLY TILTED GAUSSIAN DISTRIBUTION AND THE TILTED VAE
0.025
0.020
12)
0.005
6
0.000
-2
21
4
6
Figure Example of p+(2) when d: = 2, 3
For a tilting parameter 2 0, the exponentially tilted Gaussian distribution, denoted N6 I), the random variable on RRd= with probability density p+(2) defined by:
Definition 3.1.
(r(2) = @=1|=|
+|1=1
|||1=||
Z, Po(z), Z, = Ee-
(3)
Zr
2
Compared to the standard Gaussian, which corresponds to 7 = O. the tilting term ellll pushes the distribution towards values greater than ||>||. By completing the square, one sees the density is proportional to e ([]=1|-r)² meaning that the density is radially symmetric and has maximum value || T. An illustration provided in Figure
The tilted VAE uses the tilted Gaussian as the latent prior Pz (z). An important feature of the tilted VAE that the encoder distribution (i.e. the distribution assigned in latent space to single input a:))
For a standard Gaussian, ||ж|| is concentrated around Vd as d + oc, so the effect of tilting pushing ||*||| toward will only have an appreciable effect when > O(vd); see Figure (Right) for illustration.
4

---

Published as a conference paper at ICLR 2023
is nor exponentially tilted. We take this distribution be simple Gaussian of the form (p, I) where بر (u(z) is determined by the encoder parameters. For the sake of simplicity and to reduce computation we have fixed the covariance matrix Σ = I, whereas for the Gaussian VAE the variance I = diag(a) is allowed to be chosen by the encoder. We believe that would be possible to also allow this extra flexibility for the tilted VAE but we do not do so here ?
Intuitively, the tilted prior allows the encoder distributions to choose ht anywhere on the surface of a hyper-sphere with minimum KLD, rather than located at the single point μ O, as would be the case when the prior is an standard Gaussian. Note also that since a Gaussian cannot be perfectly fit the distribution (0, I), there will always be a non-zero minimum KLD between the encoder and prior distributions, i.c. 8(T) == inf nERd, DKL [1.I) || Nr I)) > O when T > O. This can be interpreted as the minimum average amount of information nats that each sample contains after being encoded. The decoder is then able to use this information to differentiate distributions in the latent space. This minimum value 5(7) is referred to as the "committed rate" in Razavi et al. 2019))
3.4 NORMALIZATION CONSTANT AND THE TILTED KLD BOUND
The calculation of the distribution's 's normalization constant Z, and DKL (N (1, 7) || (0, T)) is summarized below with proof's deferred to Appendix A The normalizing constant Z, satisfies
d= 1 2 2イ
1111
d= d=+1 +
3 2 2
E, =M
+ +
M
(4)
r
2
where M is the Kummer confluent hypergeometric function M (a, b, 2) En alm) the rising factorial a!n) a(a + 1). (a + n -)). The KLD can then be written as
((t) and a(n) is
DK (I.I) A (0. 7)). = log Z+ - -- VV
L 4 -1 (- ||Al13
(- 2 +
|lal1
(5)
1
2
where L is the generalized Laguerre polynomial, _ (xx = (ataa) (-,,a+1,a)) M-
During train-
ing, instead of computing (N (p, I) || (O, exactly (involving the difficult-t-ccompute
Laguerre polynomial in (5), we use the following simple quadratic approximation of the KLD,
vastly simplifying the computation while preserving the bound (T)
(K (1, N1 (0, < 2
(llæll (T)(1) +
(6)
where (²(7) = argmin, DKL (N (n,T) (O, 7)) and (T min, DKL / () (O,
is the minimum such valueF) This bound allows easy training of the tilted VAE just (2) allowed training of the standard Gaussian VAE. Note that the term (T(() is constant and can therefore be
omitted during training. The proof of (6) is deferred to Appendix A.3 Figure 5 illustrates the
difference between the exact vs approximate KI.D in (6) and shows how T/x" (T) depends on dz.
3.5 COMPARISON BETWEEN TILTED GAUSSIAN AND REGULAR GAUSSIAN
We now highlight some important differences between the tilted and regular Gaussian distribution as the dimension of the space grows. First, the volume of the space under regions of high probability density is much larger for the tilted Gaussian. Second, regions of high probability density have a much larger contribution to the total probability for the tilted Gaussian over the regular Gaussian.
To demonstrate these facts, consider a given density function p(z) and E [0, 1]. We are interested in the set of points that have a density at least times the maximum possible density. This set is denoted R. = (a p(z)= 2 e cm p(w)) In Appendix B we show that the volume of R. for the tilted
Gaussian grows exponentially faster than the regular Gaussian with respect to the dimension of the
space. The contribution to the total probability by the set R. is given by P. [R.] == J.ER_ p(z)dz.
NNte that whenever we use the standard Gaussian VAE in this paper (e.g. to compare to the tilted VAF),
we do allow the variance a to he chosen as parameter
Note that (r) is not unique and all vectors lying on the hyper-sphere with radius || are valid minima.
5

---

Published as a conference paper at ICLR 2023
For both the tilted and regular Gaussian distribution, the integral cannot be solved in closed-form. Appendix B shows a comparison between the distributions when = 0.5.
Empirically. we observe that when using the tilted VAE, far greater proportion of latent points embedded in regions of high probability density. Figure shows an example of the differences in latent spaces when using the two different priors.
REGION
STANDARD GAUSSIAN
TILTED GAUSSIAN (T = 25)
Re = (2: p2)) > cmax p(w)}
% POINTS
PROB.
VOL.
% POINTS
PROB.
VOL.
R25%
21.3 %
1.4%
417
99.9%
88.5%
30tx
R50%
6.8%
0.8%
13.1
99.5%
73.5%
2.3x
R75%
1.2%
0.01% %
0.2
95.2%
52.7%
1.5x10'
Table 1: Numerical comparison of the "size" of high probability regions for the standard Gaus- sian distribution and tilted Gaussian distribution in 10D. The rows correspond to the regions R25%, R50%, R76% where the probability density is at least 25%, 50% or 75% of the maximum probability density, R. (= p(2) m p(w)] illustrated in Figure 1 Three separate notions of "size" are compared as follows. "% POINTS": The fraction of the data set the VAE in Figure hass in the region. "PROB." The probability P[z E Re] when is chosen according the latent distribution, VVoL."; The volume of the region.
4
OUT-OF--ISTTRIBUTON DETECTION
4.1 USING THE LIKEHOOD AS AN OOD SCORE
We now focus on a particular application of VAES to demonstrate the advantages of using the tilted Gaussian prior. The task of OOD detection be described by considering training distribu- tion P. Samples that have a high probability density under the training distribution are considered to be ID, whereas samples that have low probability density under the training distribution are con- sidered to be OOD. In practice it is common to use data from different distribution Q as examples of OOD data. As we only have access to in-distribution training data, the OOD task we described case of positive unlabelled (PU) learning Jaskie & Spanias (2022) Given that the VAE is optimized to get a lower bound on the marginal log-likelhoood (I) is a natural statistic to determine a sample OOD. Samples with higher marginal log- looe would be considered to be more likely to be ID. In this work we assume that the reconstruction or likelihood term of ( takes the form of a Gaussian likelihood and we approximate the expectation the marginal likelioodd bound to be used as an "OOD score" namely:
single sample (0(222) This allows
SOOD(x) == =-| 2|1² (1|= - -1xxt)))))))
where z (0(2(1)) sample of the latent space image of the point æ. and & is the reconstructed data point from the image i.c. â ~ Pe (=(2)). We note that value of (*)() in the KLD approxi- mation (6) is not required as it is constant for all inputs. In practice, some value of SoOD (z) must be used as a threshold to determine if point is OOD or not. By sweeping through all possible threshold values we obtain the full ROC curve.
We conduct experiment comparing the tilted Gaussian vs the standard Gaussian as prior for the VAE on two OOD detection tasks. To give quantitative measure OOD classification, we examine the Receiver Operating Characteristics (ROC) curve looking at the relationship between the false positive and true positive rates as the threshold value for (7) changes. The Area Under the Surve-Receive Operating Characteristies (AUCROC) (Fawcett) 2006) is used to give a single number measuring OOD detection performance. Details regarding the full experimental settings can be found in Appendix C and the results shown in Figure
*Code available at htw.laaa...agg....o.......oooooe

---

Published as a conference paper at ICLR 2023
OOD detection on Bushion-MNISS
OOD detection on CIFAR-
Gauss: 0.0
True Posltive Rale
0.6
Gauss: 0.375
Tic. 0.999
0.0
0.2
1.0
0.6
False Positive Roin
Falso Positive Rate
Figure 3: A comparison of the ROC of VAEs trained with standard Gaussian prior and tilted prior. The left plot corresponds to models trained on the Fashion-MNIST and CIFAR-10 data sets, respectively. In both cases, the MNIST data set is used OOD data. The AUCROC metrics shown in the plot legends. In both cases the tilted Gaussian prior leads to a significant increase in performance of the OOD detection task.
4.2 THE WILL-IT-MOVE TASK FOR OUT-OF-DISTRIBUTIO DETECTION
4.2.1 MOTIVATION: ENTROPIC ISSUES WITH VARIATIONAL AUTOENCODER
As detailed in Caterini & Loaiza-Ganem 2022), using the marginal likelihood from the VAE as an OOD score has fundamental issues for low-entropy data distributions. To describe this issue theoretically, imagine we are given samples from a "true" distribution P to distinguish from an "outside" distribution Q. We train parameterized model 22 to maximize the log-likelihood of points from P. i.e. we maximize (0) Ex-p Ex~pllog po (X)) Once pa is trained, the lower bound of Po Pe (x) ) used as an OOD score; points with low log-likelln log-likelihood are to labelled from Q rather than P because
would hope that LP(0) (o) for our choice of parameters.
However, this simple bog-likeldood comparison can fail when the entropy of the "outside" distribu- tion is low compared to that of P. One can decompose the log- lilkiiiihood into two terms:
(0) = Ex (log po (X)] = - (K po) - [H,,)
where HH| P] the entropy of the distribution P. This decomposition shows that comparing the
expected log-likelihood of to that of Q gives
(p( < (Q - HPP H[ > (Q pe) (KL(P Pa).
Even if our model perfectly chosen so that DKL P || Po)) = O, is possible observe higher average log-likelihood for points from the "outsider" distribution Q when HiP > H[[Q + (RI Pa). This phenomenon means that distributions Q with low entropy, (i.e. H[Q) small) can sometimes be hard to detect using VAE model. Indeed, experiments show that detecting constant image as out distribution can counter-intuitively be quite difficult (see "Constant" row in Table 2). To deal with this problem, propose the Will-It-Move testing procedure.
4.2.2 DESCRIPTION OF THE WILL-IT-MOVE TEST
The Will-It-Move (WIM) test works by fine-tuning the parameters of the VAE to try to "push" OOD latent points away from the model prior, while keeping known points close to their original prior. Points that are truly OOD should be easily moved, while points from the original training
Note that in practice, one maximizes the empirical average over a given sample, but here will focus on the theoretical average that gets in the large data limit: lim 1 Er log -x-r (log po (X)

---

Published as a conference paper at ICLR 2023
30
30
30
20
20
20
10
10
Train
Test ID
0
Test OOD
-10
10
10
-20
-20
-2
-30
-30
-30
-20
0
20
-20 0
20
-2 0 20
Figure 4: 2D representation of the 10D latent space the Tilted Gaussian VAE during the WIM test with CIFAR-10 the ID and MNIST as the OOD. Isoradial projection is used (i.e. r2D ||1101|| see Appendix F.1 for details). Black circles are from the training set, while green x and purple triangles are unknown points to be classified from either OOD distribution IDD respectively. (Note that the true identity the test points is unknown to the algorithm.) Originally, the VAE trained only the training point maps all points to the latent distribution well. During parameter fine-tuning in the WIM test, only the truly OOD points are easily moved away to the alternative latent distribution (in this case, standard Gaussian) and can therefore be easily identified.
distribution should be hard to move away from the original prior. other words, the answer to "Will-t-Movee? is "Yes" for OOD points and "No" for ID points during the test. After the WIM- fine-tuning is completed, the same score ( is then used to label unknown points as either OOD or ID. By moving points away form the model prior, this allows the KLD term in the likelihood bound to become very large and overcome the entropy problem previously discussed.
To precisely describe the WIM task, consider training data set X drawn iid from distribution P. We suppose that we have already used to train a VAE by maximizing the log-likelihood w.r.. some given latent distribution Zx- (e.g. using the tilted Gaussian 2x N, (0, 7)). The WIM test fine-tunes the parameters of the VAE to label the identity of points from an "unknown" data set U that has samples from both the training and the OOD distribution i.e. U {ar) ] ) U { در=1 where
x' P and نه ~ Q. Note that the identity of the points in U is not known apriori.
To do this, choose different latent space distribution for the data U, denoted by Zy. In the WIM test, the model fine-tunes the parameters so that points from X stay close to Zx and points from U move to Zo in latent space. Specifically, maximize by gradient descent the objective function that is the weighted sum their log- likelihoods for some a E R:
L(Zx, +a((Z,,))
(8)
where X()x,X)= Ex'X E (xler") [logPo (x'1=)] DKL (go(aa)) Zx)) and analogously L(Zu. U) is the log- likelioel for the data set of unknown points U w.n.t. the alternative latent distribution Zu.
The first term of (8) the original training objective the VAE, and is intended keep the data set X in it's original configuration. The effect of the second term is to push the points of U toward 2 in latent space. The hope is that OOD points from Q can easily be moved to Zr,, whereas points from P are similar enough to the original data set that the first term in (8) keeps these points around Zx despite the pull from the second term. The points that we observe moving during this fine-tuning therefore identified as the OOD points (hence the name "Will-It-Move?"..
In practice, we use Zx as a tilted Gaussian and Zy as a standard Gaussian. Implementation details of the WIM test are discussed in D.I As ablation test, perform the WIM test with both Zx and Zy as Gaussian distributions with different location parameters and observed significantly better performance with the tilted Gaussian. Results from this experiment can be found in D.2
8

---

Published as a conference paper at ICLR 2023
5 RESULTS AND COMPARISON TO OTHER OOD METHODS
Table 2: AUROC comparison between OOD detection methods with Fashion-MNIST and CIFAR- 10 training distributions for various OOD sets. A larger number is better. Note that when AUROC < 0.5, (indicated by *), then "flipping labels" would improve the classifier; see F2 for discussion.
DATASET
GAUSS
IC (PNG)
IC (JPEG2000)
RATIO
REGRET
TILT
WTM
FASHION-MNIST
MNIST
0.375*
0.993
0.351
0.965
0.999
0.999
1.0
CIFAR-10
1.0
0.970
1.0
0.914
0.986
0.997
1.0
SVHN
1.0
0,999
1.0
0.761
0.989
0.98
1.0
KMNIST
0.765
0.863
0.769
0.960
0.998
0.999
NOISE
1.0
0.324
1.0
1.0
0.998
1.0
1.0
CONSTANT
0.975
1.0
0.984
0.980
0.999
0.798
1.0
CIFAR-10
MNIST
0.0*
0.976
0.0*
0.032
0.986
0.797
1.0
AASHION-MISTT
0.032
0.987
0.035
0.335*
0.976
0.688
1.0
SVHN
0.209*
0.938
0.215
0.732
0.912
0.143*
0.991
LSUN
0.833
0.348*
0.833
0,508
0.606
0.933
0.941
CELERA
0.676
0.310
0.679
0.404*
0.738
0.877
0.997
NOISE
1.0
0.042
1.0
0.851
0.994
1.0
1.0
CONSTANT
0.015
1.0
0.269
0.902
0.974
0.0*
1.0
To further validate the performance of our proposed prior, we compare the tilted prior to variety of methods that achieve top performance in OOD detection with VAEs. The methods we consider Input complexity (IC (png)), (IC (JPEG2000)) (Serrà et al., 2020), likelihood ratios (Ratio) (Ren al. 2019), and likelihood regret (Regret) (Xiao et al. 2020). A VAE with a standard Gaussian prior and log-likelihood OOD score (Gauss) is also used as benchmark. The tilted VAE and WIM test are denoted Tilt and WIM respectively. Table a summary of the results from this analysis.
From this experiment, we observe that the tilted VAE significantly outperforms the Gaussian VAE across a variety of datasets. Furthermore, the tilted VAE alone has competitive performance with the OOD based scores we compare against. When the WIM test is used with the tilted prior we ob- serve that this method matches or improves upon the compared methods all tests considered. For difficult cases such as when CIFAR- 10 is the training distribution and the OOD datasets are SVHN, LSUN and CELEBA the WIM test achieves significant performance increases over all compared methods. When looking cases where the OOD dataset is much simpler that the training distribu- tion, for example the constant dataset, we observe methods that use the log-likelihood estimate as an OOD score (Gauss and Tilt) perform poorly.
CONCLUSION
We propose generalization of the Gaussian distribution called the tilted Gaussian, and show that can be implemented in the same way as the standard Gaussian VAE with a simple change to the KLD term. It's use as a prior distribution for the VAE was motivated by showing that when the standard Guassian used as a prior, large percentage of data points encoded into regions of low probability density. We then prove that the tilted Gaussian has exponentially more volume in high probability density regions that the standard Gaussian as function of the distribution dimension. Empirically, we show that the tilted VAE encodes far greater percentage of points regions of gigh-prbbbbility density. We introduce a new OOD score for the VAE and empirically demonstrate that it is consistent improvement over the performance of recent OOD scores for the VAE. Finally, we perform an ablation study and show that the WIM test performs better when using the tilted Gaussian prior over the standard Gaussian prior.
While this work investigated the tilted Gaussian distribution in the context of OOD detection with VAEs, believe that there many applications where the tilted Gaussian is viable drop-in placement for the standard Gaussian. We encourage researchers to experiment with this distribution in other situations.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank 3 anonymous reviewers for their thorough reading of the paper, and for the many sugges- tions which improved the paper.
REFERENCES
Alexander A. Alemi, lan Fischer, and Joshua V. Dillon. Uncertainty in the variational information bottleneck. In arXiv preprint arXiv:1807.00900 00900, 2018.
Christopher Bishop. Novelty detection and neural network validation. In IEE Proceedings: Vision, Image and Signal Processing. Special issue on upplications of neurul networks., pp. 217-222, 1994.
Francesco Paolo Casale, Adrian Dalca, Luca Saglietti, Jennifer Listgarten, and Nicolo Fusi. Gaus- sian process prior variational autoencoders. In Advances in Neural Information Processing Sys- tems, 2018.
Anthony L Caterini and Gabriel Loaiza-Ganena Entropic issues in likelihood-baat ood detection. In (Still) Can't 't Believe It's Not Better! Workshop at NewrlPS 2021, pp. 21-26. PMLR, 2022.
Hyunsun Choi, Eric Jang, and Alexander A. Alemi. WAIC, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv: 2019.
Jaemoo Choi, Changyeon Yoon, Jeongwoo Bae, and Myungjoo Kang. Robust out-of-distribution detection on deep probabilistic generative models, 2021.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspher- ical variational auto-encoders. In arXiv preprint arXiv:1804.0000 2018.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint 1rXiv:161 1.02648, 2016.
Tom Fawcett. Introduction to ROC analysis. Pattern Recognition Letters, 27:861-874, 2006.
Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric Xing. Nonparametric vari- ational auto-encoders for hierarchical representation learning. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pp. 5094-5102, 2017.
M Hoffman and M Johnson. ELBO surgery: yet another way carve up the variational evidence lower bound. In Advances in Approximate Bayesian Inference, NIPS Workshop. 2016.
Kristen Jaskie and Andreas Spanias. Positive unlabeled learning. Synthesis Lectures Artificial Intelligence and Machine Learning, 16(1):2-152, 2022.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In arXiv preprint arXiv:1312.61 2014.
Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep gen- erative models. In Proceedings of The 33rd International Conference on Machine Learning, pp- 1453, 2016.
Lars Maalée, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of la- tent variables for generative modeling. H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2019.
Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, pp. 3232-3240, 2021.
10

---

Published as a conference paper at ICLR 2023
E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan. Do deep generative models know what they don't know? International Conference on Learning Representations, 2019a.
E. Nalisnick, A. Matsukawa, Y. W. Teh, and B. Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using typicality. In arXiv preprinl (ale::990..29944 2019b.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoemcoders. In International Con- ference on Learning Representations, 2017. URL http::://openneviee..eeee fermmf
id=S1jmActe
Xuming Ran, Mingkun Xu, Lingrui Mei, Qi Xu, and Quanying Liu. Detecting out-of-distribution samples via variational auto-encoder with reliable uncertainty estimation. In arXiv preprint wrXiv:2007.08128, 2021.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with Velta-VABa In International Conference on Learning Representations, 2019.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. Proceedings Machine Learning Research, 32 (2), 2014.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and national Conference on Learning Representations, 2020.
detection with weeiihood-based generative models. In Inter-
Jiaming Song, Yang Song, and Stefano Ermon. Unsupervised out-of-distribution detection with batch normalization. In arXiv preprint arXiv:1910.09110 2019.
Casper Kane Sønderby, Tapani Raiko, Lars Maalø Søren Kaae Sp nderby, and Ole Winther. Ladder variational autoencoders. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2016.
Jakub Tomezak and Max Welling. Vae with vampprior. In Amos Storkey and Fernando Perez- Cruz (eds.), Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 1214-1223, PMLR, 09-11 Apr 2018.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distributtin detection score for variational auto-encoder. In Advances in Neural Information Processing Systems, pp. 20685- 20696, 2020.

---

Published as a conference paper at ICLR 2023
A FACTS ABOUT THE TILTED GAUSSIAN
A.1 DERIVATION OF THE NORMALIZATION CONSTANT ZT
e e =11=112
(+(2)
dz
70 V2TT"
d
Z, = Ez-N(0.)) [e<lki]]
dz
Ra
2
cx xd
Ex-x(d) [e
de where x(d) = |||||
2 Tp
E [x")
n!
n=0
(d (d +
T" #1 (d (d +3) +3.... + 2-2)
n!
n2!
n even
odd
d
where 11 = [x] 2
r (
MM ( d 72
r
d+1
d (d+1 3
M
2'2'2"
r
2 : 2'
where we have used the moments of the x(d) distribution and we let M (a, b, = -
the Kummer confluent hypergeometric function and (n) a(a 1) ... (a + n - 1) the rising factorial.
A.2 EXACT FORMULA FOR THE KLD USED IN THE VAE
Define the shorthand (u) to be the KLD defined below:
f+(x) (N 1) (0,
(N)(.n(())
EEwN((a))
In
(N.((.,)(())
-11
=EwNNiad)
In
1 e= =1111 Z,e Tll=1l Tllzll
2,
-EwNN(a.)
In
=1|=11 (=,)) - 11 wlll+ +11==11)) }
In(2,) TE.-N(u..) |||=|]] + B 11 [[) |laa12
1
-I((,)) TE.~N(a.)) [||=|] + 2 ||
>n((Z,) TL (112-1)
(-1las²)
(- ||u||3
2 |1
1
2
where we have used (E~,V(..)) |||=|]]
Eixu] = L (d/2-1)
(-uml() and (000) the generalized
Laguerre polynomial.
12

---

Published as a conference paper at ICLR 2023
d
120
d, 20
60
40
2
20
Approximation
10
15
20
30
Figure 5: Left: Comparison between the exact and quadratic approximation of the KLD DRL(N (n, I) || (O, as in 6 for dg = 10, T = 15 Right: (r)|l plotted for a variety of different latent dimensions dz. The value of [x((()) was computed numerically using gradient descent from the explicit formula in (5) Note that there is a critical t value for which μ (r) be-
comes nonzero. We conjecture that the critical T Rs vd as + oo.
A.3 APPROXIMATION OF THE KLD
Let (u) E be the KLD as in 99. Note that fr depends only on the magnitude of μ4, not the direction. With = || therefore define 9r IR R by
(x) = (-(x)) = (n)) 7E TR [||= + xēi =
(10)
where é is the unit vector ह, 01.0.0...., Proposition A.1. The function 9 satisfies
27 + 2d
d"(x)= = TE
2z + x6
and in particular for 0,
g" (x) (x) 1
Proof. The formula for g goes by taking the derivative of the definition (x)) from d²
10 using
daz 7 and the elementary fact
d²
=2 +
2 2 + xē 3
dxf || xēn
2
Since this is non-negative, it is immediate that g²(xx) < as claimed.
Remark A.2. Note that the random variable appearing in the formula for is related to the non- central Beta distribution
Σ Cient1 23
X
m n
A)
E (0
2 2
(2) + 2)
Em 2?
Entm 27
13

---

Published as a conference paper at ICLR 2023
Corollary A.3. Let (T) be the minimum of fr that fr (T)(T) (4) for all μ R and g+(lur(())) S 9r (z) for all x E R. Then:
gr gr S (ar-1l((TT)] g. (1lx((T))
or equivalently:
(+( - (lall - (+)()) + (ll (+)ll)
Proof. Since (x) S and g is differentiable, by Fermat's theorem, ge (I)(=(()))) = 0. We can now integrate to find that for x > ||>||))]
/ (x)
g1(x)= (+)())
(w) dw
/ dw
$ 00+ +
(rill
MP (T)
and finally integrating again gives the desired inequality for [1la (r)]l:
9- (x(x)) = Dr (1la
+
(w) dw
(F)
S 9t (+)1) + f (T)) du
(F)]l
9r (llp* (T)l) + (x [|wx
2
fllK pllx"( (+)ll The final inequality for
The same inequality holds for a [T)]l by integrating from follows by setting = ||
B REGIONS OF HIGH PROBABILITY DENSITY FOR GAUSSIAN VS TILTED
To compute the "size" of regions where the probability density function is at least E (0, 1) times its maximum, R. {p(2) emax p(w)}. we compute for the standard Gaussian that:
1=1 2 - |=|| < 21n (t)
x 2 cp(0) = e
This shows that this region is always a sphere for the standard Gaussian, Rs = B 221n(e-T) (0). In contrast, for the tilted Gaussian we have:
2
2 cpr(T) + e ++1+)1-)) 2 20 T- / 2
< ||||| S T
2
1
This shows that the region of high probability "shell", the difference of two spheres R. B ++ 221n(c11 (0)
B (2n(er,
(0) The formula for the volume of the high dimensional sphere
Va(r)
4/2 nd from which we can compute volumes: r(g+13"
7 4/22
V VolGaussia (R.)
(21((1/)))122
T(d/2+1)
'Nte that the inner sphere is trivial unless
/21n(1/c)
14

---

Published as a conference paper at ICLR 2023
T(d/2+1 1) (1+ (2+1/1((//)) - -TT-
7/2
Volnieed (Re)
By the "curse/blessing of dimensionality", the region of high probability density is exponentially larger for the tilted prior as a function of We compute the probability of landing in the region R50% by numerically integrating the probability density on these regions below.
Integral of Probability Density under Rs0%
t 1=1
0.6
T=2
T
0.5
T=8
T=
T 32
0.4
Probability
0.3
0.2
0.1
0.0
20
30
40
50
Dimension
Figure 6: The probability P[N, (0, I) E R50%) i.e. the contribution of R50% to the total probability.
Note that 5 0 corresponds the standard Gaussian.
C EXPERIMENTAL SETTINGS
C.I MODEL STRUCTURE AND OPTIMIZATION PROCEDURE
The parameters used for the tilted prior are 30 for the MMIIST test and 25 for the CIFAR- 10 test. The WIM test uses a = 0.1 for all of experiments. For all tests we train the VAE for 250 epochs with a batch size of 64. We use the ADAM optimizer with a learning rate of 10-4 and clip gradients that are greater than 100. The encoder and decoder of the VAE is based the work done by (Choi et al. 2019) and consists 5 convolutions layers with a fully connected layer for both and σ. The decoder uses 6 deconvolutional layers and ends with a single convolutional layer. The use of fully connected encoder layers for the tilted prior is essential as the KLD a function of || which is impractical to optimize with fully convolutional model. We initialize all convolutional weights from the distribution (0, 0.2). The model details can be viewing in C. On the Fashion- latent dimensions dN 10 and for the CIFAR-10 test used d, 100. Both Likelihood Regret and Likelihood Ratio are intended to be run with categorical cross entropy loss, thus we employ this the reconstruction loss function for all comparison models.
MNIST
C.2 IMPLEMENTING METHODS
For all comparison methods IWAE with 200 samples to derive lower bound the log- likelihood. For likelihood regret, we use gradient descent on all model parameters for 100 steps in each of the tests. We use the ADAM optimizer for this process at learning rate of 1e-4, same the
15

---

Published as a conference paper at ICLR 2023
Table Details of the model architecture where b the batch size, f the number of convolutional filters, and w is the width of the square image. After each layer of the encoder and decoder a leaky rectified linear unit was used. The filter parameter was set to f = 32 for all experiments.
ENCODER
DECODER
INPUT
INPUT 2, RESHAPE TO b X 1
5 5 CONVeX STRIDE
w/4 x w/4 DECONV x2f. STRIDE
5 X 5 CONV STRIDE
x DECONV2/x2/. STRIDE 1
5 X 5 CONV x22f, STRIDE 1
5 X 5 DECONV2Á*(, STRIDE 2
X 5 CONV2/X2/, STRIDE
5 X 5 DECONV STRIDE
RESHAPE TO b d'
5 DECONV STRIDE
LINEAR FROM d' TO d, FOR μ AND o
X 5 CONV /xe> STRIDE
training procedure. We train the background model in Likelihood Ratio by setting the perturbation ratio parameter μ be 0.2 for all tests. Besides this, the background model is trained an identical format to the regular models. For input complexity we use the OpenCV implementations of the PNG and JP2 compression algorithms.
C.3 DATASETS
The datasets that are used in our experiments are MNIST, Fashion-MNIST, KMNIST, CIFAR-10, SVHN, CelebA and LSUN. We also use two synthetic datasets that we call Noise and Constant. Images from the Noise dataset
created by sampling from the uniform distribution in the range of [0, 255] for each pixel in an image, where the constant dataset is created by sampling from the uniform distribution in the range of [0, 255] where all pixels in the image have the same value. All images are resized to shape 32 x 32. Color images are converted to gray-scale for the Fashion- MNIST test by taking the first channel and discarding the other two. Grayscale images are converted to color images by taking 3 copies of the first channel. When testing with the CelebA and LSUN datasets, we use 50000 random samples from each due to the large dataset sizes.
D WILL-IT-MOVE TEST DETAILS
D.1 IMPLEMENTATION DETAILS THE WIM TEST
The WIM test was implemented by using batches that had equal parts training data, in-distribution test data, and OOD data. (We used 256 images of each type.) Our experiments used the full in- distribution test-set, which was equivalent to approximately 25% of the OOD dataset. We trained 5 epochs on each batch, then tested both datasets for OOD points. This means that the model enjoys better performance OOD points that it hasn't be tuned on, and shows effective generalization. The gradient of the training data was weighted more strongly by choosing = 0.1.
In sime-sensitive applications, it could be infeasible run the 5 epoch WIM updates on every data point given that the backward pass through the model is far slower than a forward pass. Instead, one could fine tune once only once on some data, and then run the fine tuned model for OOD detection. In table 4 we compare the relative number of images per second of the methods used in this paper. For the WIM test we include the time for the model to do epochs of fine tuning with final forward pass as well as the time for single forward pass, which could be performed after the model has already been fine tuned.
D.2 ABLATION STUDY
We perform an ablation of the WIM test, by replacing the tilted Gaussian prior with standard Gaussian prior, meaning that both Zx and Zy are Gaussian distributions. To keep the distributions separate in the latent space, set each element of the location parameter for Zu to be 3. We observe that the tilted prior consistently improves the performance of the WIM test over the standard Gaussian prior.
16

---

Published as a conference paper at ICLR 2023
Table 4: Relative runtime comparison between methods the Fashion-MNIST dataset. A larger
number is better. Experiments were run on an NVIDIA 3090 GPU with a Ryzen 3800X CPU
METHOD
IMAGES PER SECOND
WIM INFERENCE WITH PRE-FINE TUNED MODEL
97.3
WIM RUNNING EPOCHS OF FINE TUNING PLUS INFERENCE
0,498
(PNG) IC (JP2)
95.2
92.6
RATIO REGRET
49.0
2.20
Table An ablation study comparing the performance of the WIM test when VAE is trained with
Gaussian prior VAE rather than a tilted VAE. The task is OOD detection with MasSion-MNIST and CIFAR-10 as training datasets. The metric reported the AUCROC, where larger number better.
DATASET
GAUSSIAN
TILTED GAUSSIAN
AASHION-MISTT
MNIST
CIFAR-10
0.996
1.0
SVHN
0.997
1.0
KMNIST
0.999
1.0
NOISE
CONSTANT
0.998
1.0
CIFAR-10
MNIST
FASHION-MNIST
1.0
1.0
SVHN
0.940
0.991
LSUN
0.826
0.941
CELEBA
0.958
0.997
NOISE
CONSTANT
1.0
1.0
E COMPARISON TO LARGE VARIANCE GAUSSIAN
We investigate the result of using a standard VAE with a large variance Gaussian prior. The form of the prior is (O, al) where R and we vary understand the effect of OOD detection.
This larger variance also increases the region of high density, but effectively increases the "scale" of the latent space. This means separating points by adding (O, Id) part of the encoder becomes less effective (i.e. Increasing the variance of the posterior a is equivalent to decreasing the variance added to each point during encoding.) It is not surprising therefore that changing the variance does not improve the ordinary VAF. Results can be found in Table
F OTHER DETAILS
F.1 ISORADIAL PROJECTION
The isoradial projection used in Figure I and Figure map 10-dimensional space to 2-dimensional
space, Jisoradial 12 I² so that the norms of the vectors in the domain R 1o are preserved as the radius in the range R²2 i... R I | 12D = ||*| L0D for all & R10 This requirement deter- mines the radius for every point 2, the only thing left to determine is the angle 0 in 2D polar coor- dinates. In Figure and Figure we chose the angle according to the first two principle components (PCA) of the given data set. That is chose 0 to be the angle exactly that of the dimensional
17

---

Published as a conference paper at ICLR 2023
Table 6: Results comparing standard VAE with large variance Gaussian prior. The task OOD detection with Fashion-MNISS and CIFAR-I0 as training datasets. The metric reported is the AU- CROC, where larger number better. Note that when AUROC 0.5, (indicated by *), then "flip- ping labels" would improve the classifier; see F.2 for discussion.
DATASET
10
100
1000
TILTED GAUSSIAN
FASHION-MNIST
MNIST
0.0375
0.992
0.989
0.986
0.999
CIFAR-10
1.0
0.985
0.973
0.977
0.997
SVHN
1.0
1.0
1.0
0,999
0.980
KMNIST
0.765
0.913
0.838
0.883
0.999
NOISE
1.0
0.379*
0.428
0,488
1.0
CONSTANT
0.975
1.0
1.0
1.0
0.798
CIFAR-10
MNIST
0.0*
0.005
0.003
0,004
0.797
PASHION-MIIST
0.032
0.065
0.056
0.078
0,688
SVHN
0.209
0.417*
0.407
0.355
0.143
LSUN
0.833
0.836
0.823
0.823
0.933
CELEBA
0,676
0.767
0.795
0.760
0.877
NOISE
1.0
1.0
1.0
1.0
1.0
CONSTANT
0.015*
0.006
0.011
0.007
0.0*
vector (ΣPCAI, A2) of the first two principle components of z, i.c. 0 = arctan( SPCAT where
ZPCA1 is the first component of the PCA and APCA2 is the second component. These components linear combinations of the original components determined from the dataset. The projection
can hence be written as
Iisoratial (2) cos( Rsin(())
|||2|
2PCA1
.||1||
2PCA2
/2pCA1 + +22CA2 V2pCA1 + 22CA22
F.2 DISCUSSION OF CLASSIFERSS WITH AUROC < 0.5
For any classifier with AUROC 0.5, better classifier be constructed by "flipping" the classifier (i.e. when the original classifier outputs "OOD", the new classifier will label this as in- distribution and vice versa). The AUROC of the new classifier is simply 1- the AUROC of the original classifier. This is simple way to "fix" many of the poor classifiers listed in Table
For clarity, chose to only report the AUROC of the original classifier. This makes interpreting Table 2 straightforward since the exact same classifier is used in each column (as opposed to mixture of fliped/un-flipe flipped classifiers). Additionally, since the choice of whether or not to flip is evidently very data dependent, in practice one would not know apriori whether flipping improves the classification score (e.g. if classifying only a single data point, there would be no way to know). OOD classifiers that consistently output high AUROC scores across many different data sets (such as the WIM method) are therefore much more useful than classifiers that sometimes have to flipped.
In the setting of OOD for VAEs, this kind of poor performance leading to AUROC < 0.5 has been widely reported when the OOD distribution images are much "simpler" than the in-distribution images, see e.g. Nalisnick et al. (2019a). One possible way to understand this is from an information theory point of view, which is described in Section 4.2.1
18


Reviews:
[{"review_id": "Q68N3nyUS1u", "reviewer": "Reviewer_T18J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem that the high probability density region of the ordinary Gaussian prior becomes small as the latent dimension increases in VAE.\nThe authors proposed a tilted Gaussian. This distribution on the hypersphere is exponentially larger in volume than the Gaussian according to the dimension. It is easy to use because it is close to the ordinary Gaussian in terms of formulation, and we can straightforwardly implement the algorithm on top of the naive VAE.\nThey also proposed the Will-It-Move test, where they fine-tune the parameters of the VAE to improve OOD detection performance further.\nExperimental results on multiple public datasets demonstrate that the proposed method performs better than existing methods.", "summary_of_the_review": "The proposed method is reasonable, and the experiments are good.\nHowever, the comparison is only with the anomaly detection methods, and there is no comparison with the VAE variants that are proposed with the same motivation as the proposed method.", "strength_and_weaknesses": "*Strength\n- The model is easy to implement by changing the parameter definition in the naive VAE algorithm.\n- The model is reasonable to the problem they address and simple enough as it only has a single hyperparameter.\n\n*Weaknesses\n- Although the WIM test is reasonable for their purpose, it uses samples from both the training and the OOD distribution. I wonder how we can access OOD distribution. Also, if we can have access to the OOD distribution, why don't they simply use samples from the OOD distribution itself?\n- Section 2 is unclear in relation to the proposed method. Especially, Section 2.1 just lists existing methods.\n- It is better to have a more detailed discussion on and comparison with the hyperspherical VAE (Davidson et al., 2018), which also uses hyperspherical.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6545rzfP6_m", "reviewer": "Reviewer_Sn18", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers using VAEs for out of distribution (OOD) detection . The authors argue that the standard normal prior for latent variables is too concentrated to allow a proper representation of \"in distribution\" data and propose to use instead a distribution that is concentrated around a sphere in the latent space. The authors show in experiments that this \"tilted Gaussian\" prior is indeed better suited for the OOD task.  Furthermore, they  propose an even stronger approach based on fine-tuning an ensemble composed of the learned VAE and an additional VAE with a different prior on test data.", "summary_of_the_review": "The \"out of distribution\" recognition task is of high practical relevance. The paper proposes interesting and novel methods for such tasks which are based on VAEs.", "strength_and_weaknesses": "Paper strengths:\nThe authors show that learning VEAs with the proposed alternative prior is feasible and provide a justified approximation for the KL-divergence between standard Gaussians and the \"tilted Gaussian\" prior. The OOD classification is then done by thresholding the estimated likelihood.  The authors propose a second and stronger OOD classification approach via an additional fine-tuning of an ensemble of two VAEs with different prior distributions, which is interesting (even though it introduces additional run-time cost). The experiments are convincing. They show that the first proposed method (based on estimated likelihood) is competitive and that the advanced method outperforms state of the art methods based on generative models.\n\nPaper weaknesses:\n- I find the motivation given in Section 4.2.1, which claims to explain known issues of generative models for OOD detection as a consequence of low entropy distributions of OOD data, rather not convincing. I would rather assume that the \"places\" to which a VAE encoder maps OOD data is not fully predictable since such data were not used in training. It might be just the \"smoothness\" of the learned encoder/decoder mappings, that in most cases ensures that OOD data have smaller likelihood in the resulting model. \n\n- I would have expected an experimental comparison with known VAE models that use more general priors as e.g. VampPrior.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4wIpvqIgm8", "reviewer": "Reviewer_damh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents two related ideas on the topic of unsupervised density modeling for out-of-distribution detection. The area of specific focus within this topic is variational autoencoders (VAEs).\n\nFirst, they suggest an alternative prior for the latent code z of the generative model. Instead of the standard multivariate Gaussian, they suggest that p(z) should be a *tilted* Gaussian (see Defn 3.1 and Fig 2), with one hyperparameter \\tau > 0. The key idea is that in high dimensions (when d_z is large), the highest density does not occur at one point (the origin), but \ninstead on a d_z-1 dimensional subspace wherever the vector z has magnitude such that ||z|| = \\tau.\n\nTable 1 compares these two candidate distributions in terms of the *volume* and *probability mass* assigned to the space whose density (pointwise) is above 25/50/75% of the maximum density. The tilted Gaussian's set has much larger volume and probability mass than the usual Gaussian (e.g. at 25%, tilted mass is >88%, standard is below 2%).\nAn essential claim is that because the tilted prior's high-density regions are larger, there is less \"conflict\" (my term) in the ELBO objective between the expected likelihood and the KL between q and prior.... the KL term can afford to map example features to more diverse codes, rather than crowding the origin.\n\nSeveral useful facts are presented about this distribution, which (I believe) this paper is the first to propose\n* normalizing constant is given in Eq 4\n* exact KL from any Gaussian to a tilted Gaussian is given in Eq 5\n* upper bound on the KL from any Gaussian to a tilted Gaussian is given in Eq 6\nEq6's tractable bound makes training expedient, as the exact KL is expensive.\n\nSecond, the work proposes a new paradigm for training OOD detection called \"Will it Move\". The idea is to first train an unsupervised VAE (with the tilted prior) on \"normal\" training data. Given an unlabeled set U containing both \"normal\" and \"OOD\" samples, we can learn to classify the OOD ones by *fine-tuning* according to a weighted objective (Eq 8) that sums an ELBO using tilted prior on the normal training data with an ELBO using the standard Gaussian prior on the mixed U data. The intention is that OOD samples in U will freely move (in latent space) to be better explained by the Gaussian prior, while the ID samples in U will stay put (because they are similar enough to the normal data).\n\nExperiments in Table 2 examine training on Fashion MNIST / CIFAR10 then distinguishing between ID samples and OOD ones from another dataset (MNIST or KMNIST ... / SVHN or LSUN or CelebA or ...). The Will-it-Move paradigm's AUROC is above 0.94 for all dataset pairs tested, while using the VAE ELBO as a OOD score in some cases gets only 0.67 for Gaussian prior and 0.88 for tilted prior.\n", "summary_of_the_review": "Overall this paper has two ideas that both seem elegant and useful, especially the tilted Gaussian prior as a way to avoid KL collapse problems (all examples push to the origin in code space).  Technical novelty is high, ideas appear sound, experiments are convincing. I recommend accepting.\n\nI do look forward to hearing the authors' thoughts during discussion period: I think I raised a few points that will improve the paper further.\n\n", "strength_and_weaknesses": "\n# Strengths\n\n* The tilted prior is new to me and seems useful as a way to improve the volume of high-density regions\n* Results in Table 2 are convincing across many dataset pairs that the Will-it-Move idea works well for OOD\n* Comparisons to many other VAE-like approaches to OOD detection in Table 2 are welcome\n* Volume analysis in Table 1 is compelling argument for tilted prior over standard Gaussian\n\n# Weaknesses\n\nThere's several issues that I'd appreciate hearing about from the authors in rebuttal\n\n* W1: Several AUROC scores far less than 0.5 seem suspicious\n* W2: Runtime Scalability of Will-it-move test?\n* W3: Experiments only compare to other VAE models\n* W4: Perhaps compare to a Gaussian prior with larger variance?\n* W5: Exact design choices could be more clear in paper", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster