Revisiting flow generative models for Out-of-distribution detection
Authors: Dihong Jiang
Sun Sun
Yaoliang Yu
Venue: ICLR 2022 Poster (2022)
Keywords: flow models
out-of-distribution detection
random projection
distribution comparison

Abstract: Deep generative models have been widely used in practical applications such as the detection of out-of-distribution (OOD) data. In this work,  we aim to re-examine the potential of generative flow models in OOD detection. We first propose a simple combination of univariate one-sample statistical test (e.g., Kolmogorov-Smirnov) and random projections in the latent space of flow models to perform OOD detection.  Then, we propose a two-sample version of our test to account for imperfect flow models. Quite distinctly, our method does not pose parametric assumptions on OOD data and is capable of exploiting any flow model. Experimentally, firstly we confirm the efficacy of our method against state-of-the-art baselines through extensive experiments on several image datasets; secondly we investigate the relationship between model accuracy (e.g., the generation quality) and the OOD detection performance, and found surprisingly that they are not always positively correlated; and thirdly we show that detection in the latent space of flow models generally outperforms detection in the sample space across various OOD datasets, hence highlighting the benefits of training a flow model.

Published as a conference paper at ICLR 2022
REVISITING FLOW GENERATIVE MODELS FOR GROUP- WISE OUT- OUT-OF-DISTRIBUTION DETECTION
Dihong Jiang
Sun Sun
School Computer Science
School of Computer Science
University of Waterloo
University of Waterloo
Vector Institute
National Research Council sun wun.suunnre--cc... ca.ca
dihong jiangeuwaterloo.
Yaoliang Yu
School of Computer Science University of Waterloo
Vector Institute
yaolian mooliang.... woannnyyywwwttero.oooo
ABSTRACT
Deep generative models have been widely used in practical applications such the detection of out-of-distribution (OOD) data. In this work, we aim re-examine the potential of generative flow models in OOD detection. We first propose a simple combination of univariate one-sample statistical test (e.g.. Kolmogorov-Smirovv) and random projections in the latent space of flow models to perform OOD detection. Then, we propose a two-sample version of our test to account for imperfect flow models. Quite distinctly, our method does not pose parametric assumptions on OOD data and is capable of exploiting any flow model. Experimentally. firstly we confirm the efficacy of our method against state-of-the art baselines through extensive experiments on several image datasets; secondly we investigate the relationship between model accuracy (c.g.. the generation qual- ity) and the OOD detection performance, and found surprisingly that they are not always positively correlated; and thirdly we show that detection in the latent space of flow models generally outperforms detection the sample space across various OOD datasets, hence highlighting the benefits of training a flow model.
INTRODUCTION
Reliably detecting distrribbtiiinoinn (OOD) data, also known anomaly, plays important role in many deployed machine learning systems, particularly so when the system may be under at- tack by malicious adversary (Markou & Singh| 2003a Toth & Chawla 2018 Chalapathy & Chawla 2019). The bulk of existing OOD detection algorithms, in one way or another, boils down thresholding the (density) likelihood, which can be conveniently estimated by modern flow-based generative models. However, quite surprisingly, recent studies (Nalisnick et al.) 2019a) reveal that flows that are trained maximize the likelihood of in-distribution (InD) data may actually assign higher likelihood to OOD data. For example, the flow- model Glow (Kingma & Dhariwal, 2018) trained on CIFAR-11 10 that contains natural images (e.g., dog, cat, and ship) is found assign a higher likelihood to SVHN that consists of house numbers. Such surprising and counter-intuitin observations lead to natural questions on the applicability of flow models for performing the OOD detection by merely thresholding the log- likeeiood...d
In this work, we re-examine the ability flow-based generative models for OOD detection. Flow models typically transform prior distribution that is easy to sample from to the data distribution via an invertible mapping (Tabak Vanden- Eijnnenn 2010 Rezende & Mohamed 2015 Dinh et al.) this property,
propose to compare the training and test samples in the latent space of a flow model, as opposed to relying on point-wise scoring functions such the logllikel To this end, we divide test samples into minibatches (groups). Examples
2017 Kingma & Dhariwal 2018). Building

---

Published as a conference paper at ICLR 2022
of group OOD detection with generative models include for instance Nalisnick et al. (2019b); Song jet al. (2019); Chalapathy et al. (2018); Zhang et al. (2020). most of which consider either the raw input certain representation of the raw input in the sample space for detection. In contrast, exploit data representations the latent space, and confirm its advantage experimentally. Moreover, cope with the curse of dimensionality (as is typical in image datasets), we propose leverage random projections (Friedman et al., (184), which frees one from designing any extra network architectures and is computationally very efficient. Our proposed detection algorithm combines classic univariate statistical tests and random projections in the latent space of flow models.
Besides the scoring function used for detection and the representation of data, model accuracy also greatly affects OOD detection performance (Zhang et al.) [021]] Choi et al. 2018). To account for imperfectly trained flow models, we propose a two-sample version of our detection algorithm for practical use. Surprisingly, in our experiments, we confirm that model accuracy, indicated by its generation quality, may not always be positively correlated with its OOD detection performance.
We summarize contributions follows:
We propose the OOD detection algorithms GODIKS and GOD2KS for ideal and imperfect flow models, respectively, which pose no parametric assumption on the OOD data. To evade the curse of dimensionality, we propose to randomly project the latent space of flow models onto the real line and perform univariate statistical tests (such as Kolmogorov Simmoooro--Smmmov,, KS) there. Our method is computationally very efficient, requires no extra network architecture, and unifies OOD detection in both sample space and latent space.
Experimentally, 1) compare with the state-of-the-arr benchmarks on various image datasets and obtain competitive results; 2) we confirm on larger models and real datasets that model accuracy may not always be positively correlated with OOD detection performance; and 3) we compare de- tection in the sample space versus in the latent space of the flow model, and reveal the superiority and robustness of the latter, hence highlighting the potential of flow models in OOD detection.
2 OOD DETECTION WITH FLOW MODELS
In this section, we state the OOD detection problem and propose statistical tests that exploit modern flow generative models (either perfect or imperfectly trained).
2.1 OOD DETECTION: GROUP vs. POINTWISE
i.i.d. X,) ~ p be a sample from an in-distribution (InD) p. Our goal is to construct statistical test that can decide if test sample (Yi...., Ym) is from p (InD) some unknown -ftt---disttiuttionn (OOD) q. When m = 1, i.c. we examine one test sample at a time, it is often called the pointwise OOD detection (POD) while group OOD detection (GOD) refers According to Xiong et al. (20TT). group OOD can be characterized as (1) point based, where each individual point in the group anomalous, and (2) distribution based, where a group of points shows a different pattern while any single point may seem regular. Therefore, distribution based group anomaly is only detectable by GOD methods. Practical applications include detecting Higgs Bosons as group of collision events in high-energy particle physics (Muandet & Schölkopt (013)), and detecting distributed denial-of-service attacks via a group of multi-sensor networks (Chen & Yu 2016). We construct a synthetic Gaussian experiment in appendix E to illustrate this point. Thus in this work, we will focus on studying the applicability of modern flow generative models to the group OOD detection problem.
Let D
1.
2.2 FLOW-BASED GENERATIVE MODELS
A flow-based generative model simply learns a transformation T, typically diffeomorphism, that pushes latent distribution po (e.g., Gaussian or uniform) the in-distribution p. i.e.
Z Po
T(Z) p, also denoted as Rs T.Po-
In particular, when T is diffeomorphic, we have the familiar change-of-variable formula:
px(x) (o(2)/1T'(2) = (x)) / (T (x)
(2)
2

---

Published as a conference paper at ICLR 2022
200
(a) InD: ((O 0]: T) and OOD: ([1 11,1)
(b) InD: ((0, 0, and OOD: ([5, 5).
Figure I: Examples of 2-D Gaussian the latent space. In (a) and (b), left: the densities of InD and OOD; and middle and right: the density histograms of InD and OOD in each latent dimension. means input variable while z denotes latent variable.
where [2) denotes the absolute value of the determinant of the Jacobian of T. A flow model parameterizes T through neural network and estimates T by minimizing some divergence between pr (the model density) and p (the data density), based on the training data D. Once T is learned, we can easily generate new data by simply sampling Z from Po and then pushing through T, i.c. X = T(Z).
Denote X1 L X2 if they follow the same distribution. For any transformation T, it is clear that
X1 X2 T(Xi) _ T(X2),
and the converse also holds if is invertible. In fact, sufficiently regular transformation T may preserve many familiar statistical divergences D, i.e.
D(X1, X2) = D(T(()),
for instance, when is the f-divergence, such as the Kullback-Leibe (KL) and (IS) divergence, T is diffcomorphic, and X₃ and X₂ are continuous (Csiszár 1963). Another exam-
ple is the Kolmogonov-Smirno (KS) distance for real-valued X; and monotonic T:
D(1,, X2) == sup, (x) F2(x)),
(5)
where F; is the CDF of X,, as well as the Cramér-von Mises (CvM) divergence DDaling 1957
D(X,, X2) == [[F((z) Fd
Thus, for group OOD detection, we can first train invertible flow model based on the InD samples and then construct scoring function or statistical test either in the sample space (where the test sample {Y3} resides), or the latent space (where (Y,) resides). However, since both Y, and its pre-image T (Y,) are typically of high dimension and the sample size m is comparatively small, some compromise needs to be made in order to evade the curse of dimensionality, which gets even more problematic for flow models whose latent dimension
greater than input dimension (Chen)
al. 2020a] Grcić et al.] 202Ib. For instance, Zhang et al. (2020) fit multivariate Gaussian in the }) and compute analytically the KL divergence between two Gaussians. Nalisnick et al. (2019b) project Y, onto the real line using the estimated log-likeling likelihood function log.r and construct a typicality test there, while Choi et al. (2018) instead employ the Watanabe- Akaike Information Criterion (WAIC).
latent space using T
To illustrate the above idea, we compare in Figure 1 the distributions of T- irain) (Xirann) and T-1 (X Lest
in the latent space using synthetic 2-D Gaussian datasets and the flow model RealNVP Dinh al.; 2017). We observe that in the latent space the distribution of the OOD samples (i.e., T-1 (XooD-tst)) )) distinct (more so in Fig 1(b) than Fig ((a)) from the distribution of the InD samples (i.e., both T-1 (XIaa-raan)) and T-1 (XinXeDDe)s)).. Moreover, we found that the transformation T-1 can roughly keep the statistical distance in the latent space. In other words, distributions that are far away from the in in-distribttion in the sample space also tend to remain far away in the latent space.
2.3 OOD DETECTION VIA RANDOM PROJECTIONS
For perfectly trained flow model, its inverse transformation T-1 should bring the in nn-iittribttinn samples to follow (approximately) the prior distribution Po. such the commonly used standard normal. is thus natural to perform OOD detection by comparing T- (X,ea) against the prior distribution Po. whereas the InD training samples Xtaai are used to train the flow parameterized by
3

---

Published as a conference paper at ICLR 2022
projz
(a)
(b)
(c)
(d)
Figure 2: Random projections of 2-d Gaussians. proji is the projection along the axis (1,0), 0), and
proj2 is the projection along (2 - -2) For the standard Gaussian, both proji (a) and projz (b) follow (0, 1); while for the correlated Gaussian, proji (c) follows (0, 1), but proj2 (d) follows N(0, 3), instead of (0,
T. To accommodate any prior distribution Po. we propose to extend standard statistical tests such as the KS distance eq. (5) to high dimensions.
To our best knowledge, multivariate KS test has only been studied sporadically in the literature, due to the computational difficulty in enumerating the maximum in high dimensions. For instance, instance,Justell Justel er al. (1997) proposed complicated numerical procedure to compute the KS distance (incidentally also the latent space) but only for bivariate distributions. Instead, we propose to randomly project high-dimensiooal random vectors to the real line and classic KS test there. We note that similar ideas have been used to train generative models (c.g. Friedman et al.) 1984 Bonneel et al.) 2015 Kolouri al. 2016 Liutkus et al., 2019 Paty & Cuturi [219 Nguyen et al. 2021), while allow any flow models and use random projection only to construct the KS test (and related).
Figure 2 provides simple example illustrate the main idea. Consider two 2-d Gaussian distri- butions: (NO0, 0J. I) and (N[0, 0j. [1, 0.8; 0.8, 1)). Obviously, for the standard 2-d Gaussian all of its random projections along normalized directions follow (O, 1), whereas for the shown corre- lated Gaussian, only the projections along the coordinate axes follow (0, 1). Therefore, we can distinguish these two distributions by comparing their projections along random directions.
More generally, the following theorem provides the theoretical basis for distributional comparison using random projections.
Theorem Allesta-lleertos et al. 2007 Let X and Y be two Rd-valued random vectors. Suppose oc. If the set W {w
the absolute moments my := E X| are finite and Σx= (m4)) -1/k Rd www T d YY has positive Lebesgue measure, then d
The assumption Eg (m+) oc, known Carleman's condition, is very mild: it is satisfied if the underlying moment generating function is finite around the origin (hence such distributions are uniquely determined by their moments). Most distributions used in practice, such the Gaussian distribution, clearly satisfy Carleman's condition. Put differently. Theorem iimplies that a single random direction almost surely allows us distinguish the projections of X and Y; see Figure for an illustration. We note that Theorem [ can be slightly strengthened if we project to higher dimensional subspaces (Cuesta-Albertos et 2007). which, however, renders the classic KS test inapplicable. Therefore, in this work, we will only consider random projections onto the real line.
To be more specific, given a limited collection of test samples X and a flow model T, we first transform into the latent space and obtain Z = (X,a)), Then, we sample n (uniformly) random directions W Rdxx which are obtained by normalizing i.i.d. samples from the d-dimensional standard Gaussian. When the flow model T is well-trained, the inverse transformation T-1 ap- plied to InD samples will bring them to follow approximately the latent prior distribution, e.g., d-dimensional standard Gaussian. As result, projection onto each random direction yields close proximity to the univariate standard Gaussian, which classic statistical tests such as KS would be able pick up. In practice, we found that averaging over different random directions leads to slightly better and more robust performance, although the benefits quickly saturate as we increase the number of random projections. Crucially, the KS distance in eq. (5), with F, being the empirical
4

---

Published as a conference paper at ICLR 2022
Algorithm 1: Group OOD detection based on one-sample KS test (GODIKS).
Input: Test OOD samples Xee splitted into groups i..... X with each X; R (b for batch size and d for dimension); random projection matrix W Rdxx.
for i 4--11 1 to do
Z: - (X,) E Rdxb
// transform into the latent space
S W Z, E Ruxb
// project onto random directions
for j - to n do
kaj KS(S): (6)
j: 1))
// conduct one-sample KS test // average over n random directions
ly شド= kis
compute AUROC for Xes based on all k;'s
distribution of random projections W Z and F3 the latent prior distribution, can be computed in linear time by just enumerating x over each projected sample.
We call the resulting algorithm group OOD detection based one-sample KS test (GODIKS) and summarize in Algorithm [ Its computation requires one pass of the (inverse) flow model and the remaining operations are linear-time. A higher value of the KS statistics kay indicates lower similarity between the test sample and the latent prior distribution, which can be taken as a metric of ness We note that algorithm is completely general and efficient:
unlike Zhang et al. (2020) we do not require any matrix inversion or determinant and we avoid the difficult problem of estimating high dimensional covariance matrices when only very limited test samples available;
in principle, we can work with any latent prior distribution and any univariate statistical tests. As pointed out by Jaini et al. (2020), a heavier tailed latent distribution, or even discrete ones, than the standard Gaussian may be advantageous in certain settings. Similarly, other statistical tests may prove useful if we desire to zoom in certain parts of the distribution. Our choice of the KS test is motivated by our experimental settings below and serves concrete example.
our algorithm can work with any flow model T. For instance, we may even take T = Id, in which case Algorithm [ggorithm[rrduuesss reduues to performing statistical tests the sample space. Thus, our algorithm unifies the two perspectives: test in the sample space test in the latent space, which we will compare experimentally below.
2.4 IMPROVEMENT FOR IMPERFECT FLOW MODELS
When the flow model is not well-trained (perhaps even intentionally, for instance, if we take T Id), the effectiveness of the one-sample test in Algorithm | becomes questionable even for InD samples. simple fix is then to run the two-sample version of the KS test in eq. (5). where F1 is the empirical distribution of the projected test samples while F3 is now the empirical distribution of the InD training samples. More concretely, in Algorithm I we additionally derive Zerain T-2(X and project similarly to obtain Stain W Zurin Then, in Line
substitute the latent prior distribution (e.g. standard Gaussian) with the empirical distribution of the j-the row of S Below we call this modification as group OOD detection based on two-sample KS test (GOD2KS). We note that the computational complexity of GOD2KS remains similar to Algorithm [ and it is equally flexible: we can now even take snapshots of obtained during training the flow model, and run GOD2KS all of them. The detailed algorithm for GOD2KS is presented in Appendix
3 EXPERIMENTAL RESULTS
We perform extensive experiments compare proposed OOD detection algorithms with the state-of-theear (SOTA) group OOD detection benchmarks, i.e. Typicality test (TyTest) (Nalismick) Detection (KLOD) (Zhang et al.] 2020 Implementation details of benchmarks are given in Appendix C We test on two popular flow models: Glow (Kingma & Dhariwal 2018) and RealNVP (Dinh al. 2017) (see Appendix B
et al. 2019b) and the KL divergence based
5

---

Published as a conference paper at ICLR 2022
Table 1: AUROC on RealNVP (higher is better). Our results are denoted by
Highest AUROC are in boldface, and failure cases (where AUROC is below 0.5) are underlined.
batch size
batch size 10
InD
OOD
TyTest
KLOD
Ours
TyTest
KLOD
Ours
MNIST
0.77
0.95
0.99 0.99
0.82
0.96
1.00
FMNIST
KMNIST
0.97
0.97
0.99 0.99
0.97
0.98
1.00
Omniglot
1.00
1.00
10001.00
1.00
1.00
1.00/1000
SVHN
0.89
0.29
0.88
0.95
0.35
0.99
CelebA
0.67
0.90
0.76
0.81
0.99
0.89 0..90
CIFAR-I0
LSUN
0.60
0.72
0.61 (0.62
0.70
0.81
0.66|0.70
SVHN
0.92
0.23
0.89]0.84
0.98
0.29
0.99
CIFAR-100
CelebA
0.42
0.83
0.74
0.46
0.97
0.88 0..8
LSUN
0.49
0.63
0.57
0.53
0.72
0.61 00.5
CIFAR-10
1.00
0.99
0.89
1.00
0.99
0.95
CIFAR-100
1.00
0.99
0.90
1.00
1.00
0.96
SVHN
CelebA
1.00
1.00
0.92
1.00
1.00
0.98/0.97
LSUN
1.00
1.00
0.93
1.00
1.00
0.98 0.99
CIFAR-I0
0.98
0.99
0.92
1.00
1.00
0.98 00.99
CIFAR-100
0.98
0.99
0.91
1.00
1.00
0.98 00.99
CelebA
SVHN
0.78
0.80
0.97
0.81
0.99
1.00 1.00
LSUN
1.00
1.00
0.91
1.00
1.00
0.98 00.9
for more details about network architecture). For evaluation, we focus on the Area Under Receiver Operating Characteristic (AUROC/AUC) and Area Under Precison- curve (AUPR), which are commonly used in OOD detection. We compare the OOD detection performance across wide variety image datasets, including grayscale and RGB image datasets (please refer to Appendix A for more details).
3.1 ROBUSTNESS OF GOD2KS
In Table 1 we compare the performance of GODIKS and GOD2KS with the flow model RealNVP against the benchmark algorithms. For fair comparison, we adopt the same RealNVP model in the benchmarks. AUPR comparison is given in Table in appendix G.2 We use InD to denote the in- distribution dataset and OOD to denote different test datasets. To implement group OOD detection, we divide the test samples into small batches and vary the batch size for comparison. We use 50 random projections in all cases, and set as the default for all experiments on RealNVP. As expected, the performance improves with the batch size (see Appendix G.5 for results with batch size 20). We observe that, the performance of the two benchmarks and our GODIKS on these image datasets is unstable, and for some datasets the detection performance 0.5). For example. for Teses with CIFAR- 100 (InD) and CelebA (OOD), the AUROCs are 0.42 and 0.46; for KLOD with CIFAR-I0 (InD) and SVHN (OOD), the AUROCS are 0.29 and 0.35. In contrast, the performance of our GOD2KS is more robust and is generally satisfactory over all datasets. Results on Glow is shown in Table g in Appendix G.3 Again, our GOD2KS exhibits robustness over all datasets. Benchmarks also discuss detecting CIFAR-100 as OOD when training a model on CIFAR-10, and their AUROCS are around random guess. We did the same experiment and obtain similar results. The result is not surprising, since CIFAR-10 and CIFAR- 100 are similar datasets, which highlights the need to quantify OOD-ness.
be rather poor (i.e., AUROC below
3.2 RANDOM PROJECTION VS. AUTOENCODERS
When the number of random projections is less than the input dimension, we essentially perform di- mensionality reduction before comparing the distributions. Therefore, it is natural to compare with other dimensionality reduction methods such as autoencoders. To construct the benchmark, we first feed the input image an autoencoder and then use the latent code from the encoder the input in our GODIKS/GOD2KS. The only difference is that with autoencoder we now skip the step of random projections. As an example, 2019) as the autoencoder. Figure 3 showw the comparison results on FMNIST/MNIST (InD/OOD) and CIFAR- MAAAR-1//VVIT VOSSVHN using RealNVP. The batch size is fixed 10. We vary the number of random
the Latent Space Autoregression (LSA) (Abati al.

---

Published as a conference paper at ICLR 2022
1F0
KS 0001KS
C-M 0301K5
JSD GOD1KS
KS GCDOKS
-+ LSAARwaN GOOTK3
Ldd SouhtaP GOCOKS
LSAHRORROW dccooss
CvVl GOCIKS
I50 -OCOKS
RecloP GDEMKS
GOCOMS
(a) FMNIST/MNIST
(b) CIFAR- CIFAR-IO/SVHN 10/SVHN
Bokdi
Different divergences. KS and CvM test outperform JSD. A further comparison between KS and CyM is presented ap- pendix G.4
Figure 4:
Figure 3: Random projections vs. autoencoder, (a)-(b) Per- forming random projection outperforms training an extra autoencoder in two OOD detection experiments. Besides, the OOD detection performance using RealNVP saturates as the number of random projections grows beyond 50.
projections or the latent dimension from 20 to 400, and found that using random projections out- performs autoencoder with the same level of dimensionality reduction. It is possible to improve the detection performance with an autoencoder by using a more complex network than LSA. However, random projection is still appealing, as it works effectively and requires no extra networks.
3.3 EFFECT OF DIVERGENCE MEASURES
We empirically compare the KS test with the CvM test and the JS divergence (ISD). Both the KS and the CvM tests are non-parametric, while JSD requires the density to be estimated. We use the following heuristic approach to estimate the empirical density: dividing the latent values of the test batch into 20 bins, and use the normalized count in each bin as the empirical density (note that can skip this step in both KS and CvM). Therefore, the computation time for calculating JSD is much longer. For implementation, measures. As an example, in Figure 4 we show results on one challenging dataset pair CIFAR- 10/SVHN (InD/OOD) using RealNVP (results on Glow are similar). We consider two batch sizes: 10 and 20. The number of random projections is set to be 10 for all experiments. We can observe that ISD is outperformed by the other two all cases, and the performance of KS and CvM is comparable. We further compare KS and CvM on more dataset pairs for different batch sizes (see Appendix G.4). The observation is generally consistent with KS being slightly superior to CvM, especially for the smaller batch size.
the Scipy.stats library in Python to computation these
3.4 GENERATION QUALITY VS. OOD DETECTION
Intuitively, a more accurate flow model is expected to lead to better OOD detection performance. In this section, we investigate the relationship between the model accuracy and the OOD detection performance, where measure model's accuracy by its ability to generate visually realistic high- quality new images. To impose different levels of model accuracy we consider two factors: model capacity and training time. We fix the batch size 10 in this experiment.
Model capacity: Consider Glow, where K denotes the number of steps of flow in each block, T. denotes the number blocks, and denotes the number hidden channels. For comparison we trained two different Glow models and (2) a complex Glow with K = 16, L = 3, 128. Figure 5 shows the comparison of the log-liked -ilkiiiihoodd histograms and the generated images. We two log-likee ligeliioddood are similar, the generation quality is noticeably different with the complex model generating much better images. We then our OOD detection algorithms with SVHN OOD. Surprisingly, we found that the simpler Glow yields better OOD detection performance. We also observed similar results with RealNVP (see Appendix G.6 details).
CIFAR-I0: (1) simple Glow with K 3, L 3, 64,
that while the histograms of the
In appendix G.6 visualize how the generation quality and the OOD detection performance evolve with training time. We use RealNVP with 16 blocks and 512 hidden chan- nels trained CelebA. Generally, more training time leads better model, which is indicated
Training time:

---

Published as a conference paper at ICLR 2022
cifr10,, train
clfar00 train
test
clar10 tast
swhnntest
test
823
630
1.30
Log -0.30 -133
Log Ikeihood
(a)
(b)
(c)
(d)
Figure 5: Model capacity vs. OOD detection. While the density histograms estimated from the simple Glow (a) and complex Glow (b) are similar, the generation from the simple Glow (c) and complex Glow (d) are noticeably different. More surprisingly, the OOD detection performance by simple Glow (AUROCs: 0.99 0.99) is better than the complex Glow (AUROCs: 0.96)0.95).
by the generation quality. Taking CIFAR-10 OOD, observe that the detection performance improves with training time, as shown in Figure 9 However, while we have similar observations on most dataset pairs, still found some anomalies, e-g-, Glow CIFAR-IOSVHN and FM- NIST/MNIST, where the detection performance fluctuates or even declines with the training time (see Appendix G.6).
We conclude that model accuracy, as measured by the generation quality in our experiments, is always positively correlated with the OOD detection performance. These results confirm the findings of Zhang et al. (2021); Schirmeister et al. (020) on more models and more real datasets, and reveal the surprising phenomenon that misestimated model can sometimes lead better OOD detection. We present our detailed explanation and discussion in Appendix G.6
3.5 DISTRIBUTION COMPARISON IN LATENT VS. SAMPLE SPACES
Instead of transforming the data into the latent space, to the raw inputs without training a flow model, i.e. set Id. Naturally, one wonders is necessary to use flow model for OOD detection. Or, can detect in the sample space directly? Thanks to the generality of our algorithm, we are positioned to report a fair comparison below.
directly apply our detection method
When performing detection in the sample space directly with our method, we can only use the sample version due to the absence of the prior distribution. In this experiment, the batch size is fixed to be 10 and the number of random projections fixed be 50. We first run our GOD2KS (without the inverse transformation) in the sample space on image datasets, and compare with the detection results obtained in the latent space, as shown Table in appendix G.I Interestingly, we found that detecting in the latent space is generally better than the sample space, especially for SVHN/CIFAR- 10, SVHN/CIFAR-100 and SVHN/CelebA.
Next, we examine the detection performance by varying the OOD distribution. To this end, we manually inject zero-mean Gaussian noise to the OOD data. The results are summarized in Table 4 in appendix G. We can that the detection performance in the sample space varies with the OOD distribution, while that in the latent space is superior and stable. We hypothesize that the robustness of detection in the latent space be attributed the regular structure of the latent prior distribution (i.e., Gaussian), which can be beneficial for distributional comparison.
RELATED WORK
OOD detection has been explored from different perspectives. c.g., discriminative methods, gener- ative models, or hypothesis tests; see Toth & Chawla (2018); Chalapathy & Chawla (2019); Pang et al. (2021) for extensive reviews. Below only discuss papers on unsupervised methods using generative models.
Unsupervised group OOD detection Nalisnick et al. (2019) propose the typicality test for group OOD detection under the hypothesis that the in in-distribution samples are drawn from the typical set of the data distribution, which may not overlap with the regions of high density. This hypothesis is
8

---

Published as a conference paper at ICLR 2022
recently interrogated by Zhang et al. (2021). who instead attribute the failure of OOD detection in deep generative models to model misestimation. Moreover, Song et al. (2019) observe that batch normalization can lower the likelihood of a batch of OOD samples. based on which permutation test is proposed for group OOD detection. Chalapathy et al. (018)) define a group reference func- tion which aggregates the information of input groups and then suggests a distance score for OOD detection that measures the deviation between a test group and the group reference function. It's worth mentioning that both Song al. (2119) and Chalapathy et al. (2018) adopt fixed and large batch size (64 and 1536, respectively) for all experiments, and thus it is not clear how the batch size (especially the smaller ones) would affect the detection performance. Our work is most related to Zhang et al. (2020), who also consider detection in the latent space of flow models. The main differences lie three aspects: (1) they impose the Gaussian assumption on the latent distribution of test samples, so that they can compare it with the latent prior in a closed form using KI.-divergence.
In contrast, our KS test is non non- parametric. hence does not require any distributional assumption on test data; (2) they assume the trained flow model is perfect, while we additionally provide GOD2KS for imperfect models; and (3) we use random projections to consider both marginal distribution and correlations of all dimensions, while they need explicitly estimate the correlation coefficients to quantify the inter-dimensional correlations, which can be challenging in practice.
Unsupervised point OOD detection Compared to group OOD detection, point OOD detection treats test samples individually. In the context of deep generative models, many researchers have tried to explain the failure of log- log-likelihood for OOD detection from different perspectives, e.g., the background statistics (Ren et al. 2019). the inductive biases (Kirichenko et al. 20020), the input complexity (Serrà al. 2020 and the model accuracy (Choi et al., 2018), and proposed new metrics therein. Our work can also be extended to point detection by artificially creating a batch of samples for each individual test sample (see Appendix [ more details). Note that [Schirrmeister et al. (2020); HHvtoor et al. (2021) also develop detection scores based on (partial) latent variables of flow model and VAE, respectively. Differently, |Havtorn et al.] (202T) interpret their likelihood ratio test comparing divergence between true and approximate posterior in latent space (since true posterior is intractable), whereas |Schirmeiiser et al. (2020) contrast the log likelihood of the last-scale latent code that encodes global object-specific features with that of some background latent code. Nevertheless, both approaches do not employ random projections or KS tests as we do. |Moningstar al. (202T) extended the typicality test to the pointwise setting, by fitting kernel density estimator to each of a few test statistics on the training set and then computing the sum of log probability of each statistic the detection score for a given data point.
Representation learning for OOD detection OOD detection is challenging in high dimensions. This motivates the study of representation learning, which aims characterize discriminative in- formation of the input with a reduced dimension for the OOD detection. Example methods include projection (Pevný) 2016 Pang et al.) 2018), feature selection (Azmandian et al., 2012, Pang et al. 2017), and deep learning based methods. For the latter one, representations can be obtained either from a pre-trained model (Zhou et al.] 2019; Pang et al. 22000] Andrews et al., 2016 Tudor lonescu et al. (017)), or be directly learned with a neural network such as autoencoders (Xu et al. 2015 lonescu et al.] 2019; Erfani et al.) 2016 Wang et al. 2019). In our work, the representation is ob- tained by random projecting the latent space of flow models. Compared with other deep learning based approaches, our method requires no additional networks and is computationally more efficient.
5 CONCLUSION
In this work, we re-examined the potential of flow models for OOD detection. We provided practical OOD detection algorithms that compare distributional information in the latent space and impose no parametric assumption. We compared with SOTA benchmarks and obtained comparable and more stable performances. Experimentally, we demonstrated the benefits of detecting OOD data in the latent space and confirmed that OOD detection performance is not always positively correlated with model accuracy. In the future, we would like to characterize OOD distributions that can be reliably detected by our methods, as well as the possibility of projecting onto high dimensional subspaces.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank the reviewers as well as the area chair for thoughtful comments. We gratefully acknowl- edge funding support from NSERC, the Canada CIFAR Al Chairs program, NRC and University of Waterloo. Resources used preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
REFERENCES
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481 2019.
Jerone Andrews, Thomas Tanay, Edward Morton, and Lewis D Griffin. Transfer representation-
learning for anomaly detection In International Conference on Machine Learning, 2016.
Fatemeh Azmandian, Ayse Yilmazer, Jennifer Dy, Javed A Aslam, and David Kaeli. Gpu- accelerated feature selection for outlier detection using the local kernel density ratio In IEEE 12th International Conference on Data Mining, pp. 51-60, 2012.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sllieed and Radon Wasserstein barycenters of measures) Journal of Mathematical Imaging and Vision, 51:22 45, 2015.
Raghavendra Chalapathy and Sanjay Chawla.
Deep learning for anomaly detection: A survey
arXiv:1901.03407, 2019.
Raghavendra Chalapathy, Edward Toth, and Sanjay Chawla. Group anomaly detection using deep
generative models In Joint European Conference on Machine Learning and Knowledge Discov- ery in Databases, pp. 173-189, 2018
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation In International Conference on Machine Learning, pp. 1660-1669, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations) In International conference on machine learning, pp. 1597-1607, 2020b.
Xiaofan Chen and Shunzheng Yu. A collaborative intrusion detection system against ddos for sdn IEICE Transactions on Information and Systems, 99(9):2395-23344 2016.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection arXiv:1810.0139) 2018.
Tarin Clanuwat, Mikel Bober- Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, NeurIPS Workshop on Machine Learning for Creativity and Design, 2018.
Imre Csiszár. Eine informationstheorrtishhe ungleichung und ihre anwendung auf den beweis der ergodizität von markoffschen ketten, A Magyar Tudományos Akadémia Matematikai Kutató Intézetének közleményei, 8:85-108, 1963.
Juan Antonio Cuesta-Albertos, Ricardo Fraiman, and Thomas Ransford. A sharp form of the cramer-wold theorem Journal of Theoretical Probability, 20(2):201-209, 2007.
D. A. Darling. The Kolmogorov-Smiror, Cramér-von Mises tests Statistics, 28(4):823-38, 1957.
The Annals of Mathematical
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP In International Conference Learning Representations, 2017.
Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using a linear one-class svm with deep learning
Pattern Recognition, 58:121-134, 2016.
10

---

Published as a conference paper at ICLR 2022
Jerome H. Friedman, Werner Stuetzle, and Anne Schroeder. Projection pursuit density estimation Journal of the American Statistical Association, 79(387):599 608, 1984.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George B Moody, Chung-Kang Peng, and Eugene Stanley, Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals circulation, 101(2))ee11- 2000.
Matej Grcié, Ivan Grubišić, and Siniša Segvić. vrXiv:2106.04627, 2021.
Densely connected normalizing flows.
Jakob D. Drachmann Havtorn, Jes Frellsen, Soren Hauberg, and Lars Maaløe. Hierarchical vaes know what they don't 't know In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 4117-4128, 2021.
Yuhei Ichimaru and GB Moody. Development of the polysomnographic database on cd-rom Psy- chiatry and clinical neurosciences, 53(2):155-177, 1999.
Radu Tudor Ionescu, Fahad Shahbaz Khan. Mariana- Georgescu, and Ling Shao. Object- centric auto-encoders and dummy anomalies for abnormal event detection in video, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7842-7851, 2019.
P. Jaini, I. Kobyzev, Y. Yu, and M. Brubaker. Tails of Lipschitz triangular flows In International
Conference on Machine Learning, 2020.
Ana Justel, Daniel Pe na, and Rubén Zamar. A multivariate Kolmogorov-Smirmov test of goodness of fit) Statistics & Probability Letters, 35(3):551-25 1997.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions In Advances Neural Information Processing Systems, volume 31, 2018.
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out- of-distribution datal In Advances in Neural Information Processing Systems, volume 33, 2020.
Soheil Kolouri, Yang Zou, and Gustavo K. Rohde. Sliced wasserstein kernels for probability distri- butions In CVPR, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images 2009.
Brenden Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction| Science, (3006266):13322-2 1338, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. tradienn-based learning applied to document recognition) Proceedings of the IEEE, 86(11):228--3324 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild
In IEEE international conference on computer vision, pp. 7738-3738, 2015.
Antoine Liutkus, Umut Simsekli, Szymon Majewski, Alain Durmus, and Rabian-Robert Stöter. Sliced- Wasserstein flows: Nonparametric generative modeling via optimal transport and diffu- sions In Proceedings of the 36th International Conference on Machine Learning, pp. 4103-4113, 2019.
Markos Markou and Sameer Singh. Novelty detection: a review part 1: statistical approaches
Signal processing, 83(12):2481-2497, 2003a. 2003a.
Markos Markou and Sameer Singh. Novelty detection: a review part 2:: neural network based
approaccess) Signal processing, 83 (31122224992252 2003b.
Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pp. 3232-3240, 2021.
11

---

Published as a conference paper at ICLR 2022
K Muandet and B Schölkopf. One-class support measure machines for group anomaly detection, In 29th Conference on Uncertainty in Artificial Intelligence, pp. 449 458, 2013.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do
deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019a.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs deep generative models using typicality) In NeurlPS workshop on Bayesian Deep Learning, 2019b.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing digits in natural images with unsupervised feature learning In NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional wassertessnn and applica- tions to generative modeling In International Conference on Learning Representations, 2021.
Guansong Pang, Longbing Cao. Ling Chen, and Huan Liu. Learning homophily couplings from non-iid data for joint feature selection and noise-resilient outlier detection, In International Joint Conference on Artificial Intelligence, pp. 2585-2591, 2017.
Guansong Pang, Longbing Cao, Ling Chen, and Huan Liu. Learning representations of ultrahigh- dimensional data for random distance- based outlier detection In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2041 2050, 2018.
Guansong Pang, Cheng Yan, Chunhua Shen, Anton van den Hengel, and Xiao Bai. Self-trained deep ordinal regression for end-to-end video anomaly detection In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
François-Pierre Paty and Marco Cuturi. (Susspace robust Wasserstein distances In Proceedings of the 36th International Conference on Machine Learning, pp. 5072-5081, 2019.
Tomáš Pevny. Loda: Lightweight on-line detector of anomalies 304, 2016.
Machine Learning, 102(2):275-
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark DePristo, Joshua V Dillon, and Balaji Lakshminarayanan. Likelihood ratios for dittrf--iistriution detection In Advances in Neural Information Processing Systems, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In Interna- tional conference on machine learning, pp. 1530-1538, 2015.
Robin Schirmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features In Advances in Neural Information Processing Systems, volume 33, 2020.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input
complexity and out-of oft-of- distribution detection with selihood-baseee generative models] In Inter- national Conference Learning Representations, 2020.
Connor Shorten and Taghi M Khoshgoftaar, A survey on image data augmentation for deep learning Journal of Big Data, 6(1): 2019.
Jiaming Song. Yang Song. and Stefano Ermon. Unsupervised distr--disttbuution detection with batch normalization, arXiv:1910.0911 2019.
Esteban G. Tabak and Eric Vanden Vanden-Eijnden. Density estimation by dual ascent of the log-like likelihood Communications in Mathematical Sciences, 8(1::217-233, 2010.
12

---

Published as a conference paper at ICLR 2022
Edward Toth and Sanjay Chawla. Group deviation detection methods: survey Surveys, 51(4):1-38, 2018.
ACM Computing
Radu Tudor Ionescu, Sorina Smeureanu, Bogdan Alexe, and Marius Popescu. Unmasking the ab- normal events in video In IEEE International Conference on Computer Vision, pp. 2895-2903,
2017.
Jingjing Wang, Sun Sun, and Yaoliang Yu. Multivariate triangular quantile maps for novelty detec-
tion In Advances Neural Information Processing Systems, volume 32, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- a novel image dataset for benchmark ing machine learning algorithms LcXiv:1708.07747, 2017.
Liang Xiong, Barnabas Poczos, Jeff Schneider, Andrew Connolly, and Jake VanderPlas. Hierarchi- cal probabilistic models for group anomaly detection In International Conference on Artificial Intelligence and Statistics, volume 15, pp. 789-797, 2011.
Dan Xu, Elisa Ricci, Yan Yan, Jingkuan Song, and Nicu Sebe. Learning deep representations of appearance and motion for anomalous event detection, In Proceedings of British Machine Vision Conference, 2015.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of scage-scale image dataset using deep learning with humans in the loop arXiv:1506.0336. 2015.
Lily Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-of-distribution detection with deep generative models In International Conference on Machine Learning. pp.
12427-12436, 2021.
Yufeng Zhang, Wanwei Liu, Zhenbang Chen, Ji Wang, Zhiming Liu, Kenli Li, and Hong- mei Wei. Oustoo-distributtion detection with distance guarantee in deep generative models arXiv:2002.03328w3, 2020.
Joey Tianyi Zhou, Jiawei Du, Hongyuan Zhu, Xi Peng, Yong Liu, and Rick Siow Mong Goh. Anomalynet: An anomaly detection network for video surveillance, IEEE Transactions on Infor- mation Forensics and Security, 14(10)2537-2550, 2019.
13

---

Published as a conference paper at ICLR 2022
A DATASETS
In this section, we briefly introduce the public datasets we use and our preprocessing in this work. Note that image size is represented in #channels x height width, and all images are normalized into [0.0, 1.0].
Grayscale image datasets:
MNIST (LeCun et al.
1998): MNIST is a dataset of handwritten digits, including 10 classes
(from digit O to 9) and
70000 images in total. Each image is in 1 28 28.
FMNIST (Xiao et al. of clothes and shoes. There are 70000 images in total, and each of them is in 1 x 28 x 28, 2018): KMNIST is dataset of Japanese characters. There are 10 x 28 x 28.
2017]: FMNIST is a dataset of Zalando's article images with 10 classes
KMNNSTT (Clanuwat et al.
character labels and 70000 images in total, and each of them
-Omniglot (Lake et al. 2015): Omniglot is a dataset of handwritten digits from different alphabets. There are 1623 different characters and 32460 images in total, and each of them is in 1 x 105 x 105. We center-cropped images to the size of x 84 x 84, and then resize
them into 28 28.
RGB image datasets:
CIFAR- 10/100 (Krizhevsky al. 2009): 1IFAR-10/100 are datasets of natural images, e.g.
animals and vehicles. CIFAR-10 10 contains 10 classes, while CIFAR-100 contains 100 classes.
Both datasets contain total of 60000 images. Each image is in 3 32 x 32.
SVHN (Netzer et al. 201T): j: SVHN is dataset of street view housing numbers, including
10 classes and 99289
images in total. Each image is in 3 x 32 x 32.
LLSUN (Yu et al. 2015:: LSUN is dataset of 10 scene categories, e.g., bedroom, classroom, kitchen. Only 10000 test images are used in this work. Images are not in an universal size. The smaller dimension is resized to 256 by the original dataset. We center-cropped images to the size of 3 x 224 x 224, and then resize them into 3 x 32 x 32.
Cellbb (Liu et al. 2015): CelebA is dataset of face images of celebrities. It contains total of 10177 identities and 202599 face images. Each image in 3 x 178 x 218. We center- cropped images to the size of 3 x 178 x 178, and then resize them into 3 x 32 x 32. 50000 images are randomly sampled from predefined training split as the trainig set in this work.
We use the official training and test split for all datasets, and we create the validation set by randomly holding out 10% from the training split.
B IMPLEMENTATION DETAILS ON OUR METHOD
Glow: Our Pytorch implementation of Glow was derived from Joost van Amersfoort's 's repository our main results, for grayscale image datasets, we use 2 blocks, where each block consists of 3 steps of flow with 64 hidden channels. For RGB image datsets, use blocks, where each block consists of steps of flow with 64 hidden channels. Affine coupling layers with squeezing factor of 2 are used for all datasets. All priors set as standard Gaussians. The learning rate is set as le-5 for grayscale image datasets and le-3 for RGB image datasets. The optimizer is Adam with weight decay of 1e-6.
ReaINVP: Our Pytorch implementation of RealNVP was derived from Ilya Kostrikov's repos- itory 22 In our main results, for grayscale image datasets, we use one block with 512 hidden channels. For RGB image datsets, use block with 2048 hidden channels. All priors are set as standard Gaussians. The learning rate is set as le-6 for grayscale image datasets and le-5 for RGB image datasets. The optimizer is Adam with a weight decay of le-6.
As mentioned earlier, the training epochs can significantly affect the OOD detection performance, and we found that the two SOTA baselines, as well as our GODIKS, are more sensitive to training time than GOD2KS. We use validation set to select the training epoch that optimizes our GOD2KS
14

---

Published as a conference paper at ICLR 2022
Algorithm 2: Group OOD detection based on two-sample KS test (GOD2KS),
Input: Training InD samples Xaain E R (N denotes the size of training set), Test OOD samples Xieu splitted into m groups ",, Xm with each X, Rdxb (b for batch size and d for dimension); random projection matrix W E R
Zurain -T (iinn)) R
Smain W Zurann REXN
for i to do
Z; T=x(X,)) E Rdx6
// transform into the latent space
((1) W Z, E R
// project onto random directions
for j < to n do
kij KS(S)) J: srant()))
(
conduct two-sample KS test // average over n random directions
h; 25=1 kij
compute AUROC for X based on all k
performance. Specifically, for RealNVP, we found that training the model for 50 epochs on FMNIST, CIFAR- 100, CelebA, and 100 epochs CIFAR-I0, SVHN, achieves better GOD2KS results. For Glow, we found that training the model for epochs on CIFAR-10 10, 5 epochs on CIFAR-100, and 200 epochs FMNIST, SVHN, CelebA, achieves better GOD2KS results.
We also present the detailed algorithm of our GOD2KS for completeness, as shown in Algorithm
C IMPLEMENTATION DETAILS OF BENCHMARKS
For TyTest we follow their Algorithm 1 except the threshold to calculate the detection score, since using AUROC for comparison. For KLOD, follow their equation (4) to calculate the detection score. However, we observe one empirical issue of KLOD, i.e. the estimation of a high-dimensional covariance matrix from small batch mostly singular, which leads the log determinant of covariance matrix in equation (4) of Zhang et al.) (2020) to be negative infinity, and thus not indicative as a detection score at all. In our experiment, calculate μ replicate their results.
D FRAMEWORK
Our framework is illustrated in Figure to obtain the latent variable, i.c. 2 = (I). then do the detection based on the distribution of z.
When the flow model T- is trained, we do the inference
E MOTIVATION FOR GROUPWISE DETECTION
As mentioned in section 2. distribution based group anomaly only detectable by groupwise de- tection methods. We conduct synthetic experiment to illustrate this point. Consider two 2-d Gaus- sians with the same zero mean and orthogonal cigenvectors, e.g. InD=N(([o, 0), [[1. 0.5]. [0.5, 1]]) and OOD=N 0), [[1, -0.5], [-0.5, [-0.5, 1]]). The two distributions have a large overlap around origin where the majority of mass reside in (they have the same density peak). Therefore, if we train generative model (e.g. RealNVP in this example) to estimate their density, the InD and OOD den- sity histograms will largely overlap, as shown in Figure 7 which results in an AUROC of pointwise likelihood detection score low 0.57. In contrast, with increasing number of test samples drawn from these two distributions, the separation InD and OOD will be increasingly clear (which true for both sample space and latent space). Our GODIKS and GOD2KS methods achieve detection AUROC 0.67 0.67 for batch size 10, 0.77 0.77 for batch size = 20, and 0.97 for batch size 100, respectively.
15

---

Published as a conference paper at ICLR 2022
Withou: muloencoder:
loss log po
Flow model 1r
Latent variable
Ingut
Code
Encoder
Dacoder
With autoencoder:
Reconstruction
loss log po (1r) + 1|x x
Figure 6: The framework of our method.
InD iD_train
InD_test
0.150
OOD_ test
2
0.125
0.100
0.075
0.050
0.000
14 12
-4
-2
Xo
Log likelihood
(a) InD: \(oo 0), [11, 0.5), 0.5, 1])) and OOD: (N[0, 0) [[1, -.5), 0-5,, 1]])
(b) Log likelihood
Figure 7: An example of distribution based group anomaly. (a) Two 2-d Gaussians with similar density and large overlap in support. (b) Density histograms largely overlap, where likelihood score only achieves detection AUROC of 0.57.
F POINT OOD DETECTION VIA DATA AUGMENTATION
In GOD, we assume that a group of test samples are either from the in-distribution or not. Now we will relax this assumption and extend the proposed group OOD detection point OOD detection.
The key of the extension is to obtain a group samples based on one single test sample. To do so, one option is to adopt data augmentation (Shorten & Khoshgoftaar 2019). which has been exten- sively used in deep learning when the number of the training samples is limited. Data augmentation is also common practice construct the positive pairs in contrastive learning (Chen et al. 2020b). Concretely, for each image sample we can construct a group of samples by applying, e.g., rota- tions, random cropping and resizing, and color distortion. For comparison, produce 10 and 40
16

---

Published as a conference paper at ICLR 2022
samples respectively. Specifically, 10 augmented samples consist of 3 rotated images (90, 180, 270 degrees), random cropped and resized images, and 3 color distorted (jitter) images, as well as the original one, while 40 augmented samples consist of rotated images (90, 180, 270 degrees), 18 random cropped and resized images, and 18 color distorted (jitter) images, as well as the original one. Generally, our detection performance improves with the number of data augmentation.
We compare the performance our point OOD algorithm with two SOTA benchmarks: log ((x) and input complexity S (Serrà al., 2020), shown in Table 2 Both log q(x) and S encounter complete failure detection case, e.g. CIFAR-10/VVI for log g(x) and FMNIST/KMNIST for S,
while our POD is more robust to different dataset pairs.
Table 2: AUROC on RealNVP for point OOD detection (higher is better). Our results are denoted by RI/R2, where RI corresponds GODIKS, and R2 corresponds GOD2KS.
InD
OOD
log (x)
S
Ours-10
Ours-40
MNIST
0.80
0.43
0.74
0.79
FMNIST
KMNIST
0.91
0.19
0.84 0.83
0.90
Omniglot
1.00
1.00
1.00 1.00
1.00|
SVHN
0.16
0.80
0.85
0.85
CelebA
0.65
0.51
0.59 0.59
0.57 0.57
CIFARIO
LSUN
0.60
0.48
0.45 0.45 0.47
0.46 0..48
G ADDITIONAL RESULTS
G.1 SAMPLE SPACE VS. LATENT SPACE
Experimental results of detecting in sample space and latent space are summarized table and table 4 with injecting noise to OOD data).
G.2
GOD RESULTS USING REALNVP AUPR
The corresponding AUPRs for the experiments in Table are shown in Table
G.3 GOD RESULTS USING GLOWW
The optimal number of random projections can be dataset- and model-dependent, We run a similar experiment as section 3.2 with Glow to select the optimal number of projections. The selection process is similar to section 3.2 thus is omitted. Finally, we use 100 random projections when training Glow on grayscale image datasets, and use 200 random projections when training Glow on RGB image datasets. The detection result is summarized in Table Comparing Table 6 with Table we can notice that the detection performance can be largely affected by the network architecture. In particular, for KLOD and our GODIKS the AUROC for FMNIST (InD) and MNIST/KMMNIST (OOD) can be close to O, which indicates a complete failure of detection. The underlying reason that it is always questionable that trained flow model is pperfect', which means there always possibility that T- sample version to improve the robustness.
#OOD is closer to the latent prior than T-1
#InD motivating us to propose the two-
G.4 COMPARISON OF DIFFERENT DIVERGENCE MEASURES
We conduct few more experiments to compare KS test and CvM test, shown in Table 7 where the KS test indicates more robustness especially when batch size small.
G.5 BATCH SIZE = 20
We add more results when batch size is 20, shown in Table & The model is the same Table | and 6 As expected, the GOD performance will increase as batch size increases. Our method can achieve AUROC of 1.00 for most cases when batch size is 20.
17

---

Published as a conference paper at ICLR 2022
Table 3: Comparison in the sample and latent spaces. AUROCs are shown and segmented by |. denotes detection in the sample space, while L-R and L-G denote detecting in the latent space us- ing RealNVP and Glow, respectively.
Table Comparison in the sample and latent spaces with noise. AUROCs are shown and segmented by S denotes detection in the sample space, while L-R and L-G denote detecting in the latent space using Real- NVP and Glow, respectively.
InDIOOD
Noise
S L-R L-G
InD/OOD
$
L-
L-G
No noise
0.90
0.97 0099
FMNIST/MNIST
1.00
1.00
1.00
(00, 0.1)
0.86 1.00 1.00
CIFAR- 10/SVHN
0.90
0.97
0.99
CUFAR-IOSVVII
[ 0.2)
0.90 1.00 1.00
CIFAR- 10/CelebA
0.96
0.93
0.99
0.98
1.00 1.00
SVHN/CIFAR-ID 10
0.62
0.98 0.99
(0,40.4)
1.00 1.00 1.00
SVHN/CIFAR- 100
0.67
0.98 0.99
No noise
0.62
0.98 0.999
SVHN/CelebA
0.88
0.99
1.00
0.1)
0.69 1.00 1.00
SVHNICIFAR-10
0.2)
0.79 1.00 1.00
0.3)
0.88
1.00 1.00
0.4,
0.93
1.00 1.00
Table 5: AUPR on RealNVP (higher better). Our results are denoted by GODIKS|GOD2KS, Highest AUPR are in boldface, and failure cases (where AUPR is below 0.5) are underlined.
batch size 5
batch size = 10
InD
OOD
TyTest
KLOD
Ours
Ty Test
KLOD
Ours
MNIST
0.65
0.86
0.99
0.68
0.89
1.00 1.00
FMNIST
KMNIST
0.95
0.95
0.99 0.99
0.96
0.96
1.00 1.00
Omniglot
1.00
1.00
1.00 1.00
1.00
1.00
1.00 1.00
SVHN
0.95
0.68
0.95
0.98
0.72
1.00
CelebA
0.78
0.94
0.86
0.87
0.99
0.94
CIFAR-10
LSUN
0.60
0.67
0.60/0.61
0.70
0.78
0.76
SVHN
0.96
0.63
0.96
0.99
0.68
1.00
CIFAR-100
CelebA
0.60
0.89
0.85
0.63
0.98
0.93
LSUN
0.49
0.57
0.56
0.52
0.66
0.60
CIFAR- 10
1.00
0.98
0.62
1.00
0.98
0.80
CIFAR- 100
1.00
0.98
0.64
1.00
0.99
0.82 0.90
SVHN
CelebA
1.00
1.00
0.79
1.00
1.00
0.93
LSUN
1.00
0.99
0.70J0.74
1.00
1.00
0.93
CIFAR -1
0.97
0.98
0.86
1.00
1.00
0.97
CIFAR -IFAR-100
0.98
0.98
0.85
1.00
1.00
0.96
CelebA
SVHN
0.83
0.85
0.98
0.86
0.99
1.00
LSUN
0.99
0.99
0.85
1.00
1.00
0.96
G.6 GENERATION QUALITY VS. OOD DETECTION
Model capacity: We repeat the experiment regarding generation quality on RealNVP, and the comparison is shown in Figure g Again, the batch size is fixed as 10. The observation is similar to Figure 5 i.e. a less capable model that generates more blurred images performs better in OOD detection.
Training time:
While the OOD detection performance generally improves with training time, as shown in Figure g still found some anomalies. We found few counterexamples that the OOD detection performance fluctuate or even declines with training time, as shown in Figure 10 and
This counter-intuitive phenomenon might be surprising at first glance, but it is not completely baseless. Note that the OOD detection failure for FMNIST/MNIST and DLFAR-10/SVER (the two most famous OOD detection failure cases reported by Nalisnick al. (019aa) is because the generative model gives OOD input higher density estimation than InD, L.c. the log likelihood histogram of OOD sits the right hand side of InD. A better model (either with larger capacity
18

---

Published as a conference paper at ICLR 2022
Table 6: AUROC and AUPR on Glow (higher better). Our results are denoted by GODIKS/GOD2K Highest metrics are denoted in boldface, and failure cases (where metric is below 0.5) are underlined.
hatch size 5
batch size 10
InD
OOD
Metric
TyTest
KLOD
Ours
TyTest
KLOD
Ours
AUC
0.99
0.12
0.19
1.00
0.04
0.11
MNIST
FMNIST
AUPR
0.32
0.34
1.00
0.31
0.32 1.00
AUC
0.51
0.38
0.01 0.97
0.55
0.13
0.00
1.00
KMNIST
AUPR
0.53
0.41
0.31
0.56
0.33
0.30 1.00
AUC
1.00
1.00
1.00
1.00
1.00
1.00
Omniglot
AUPR
1.00
1.00
1.00
L.00
1.00
1.00
AUC
0.96
0.98
T.00)
1.00
1.00
SVHN
AUPR
1.00
0.98
0.99/0.98
1.00
1.00
1.00
CIFAR-10
AUC
0.55
0.98
0.76
0.68
1.00
0.79
CelebA
AUPR
0.69
0.99
0.92
0.77
1.00
0.88 0..9
AUC
0.86
0.82
0.61
0.94
0.80
0.18
LSUN
AUPR
0.81
0.38
0.95
0.78
0.33
AUC
1.00
0.79
0.990
1.00
0.97
0.99
SVHN
CIFAR-100
AUPR
1.00
0.91
0.98
1.00
0.99
0,99
AUC
0.35
0.97
0.74
0.35
1.00
0.89
CelebA
AUPR
0.55
0.98
0.90
0.55
1.00
0.91 0.98
AUC
0.82
0.38 0.59
0.76
0.86
0.27 0..59
LSUN
AUPR
0.66
0.80
0.42 0.58
0.78
0.85
0.37 0.58
AUC
1.00
1.00
0.10
1.00
1.00
0.01 00.99
CIFAR-10
AUPR
1.00
1.00
0.16
1.00
1.00
0.15 0.98
SVHN
AUC
1.00
1.00
0.08 0.93
1.00
1.00
0.01 0.99
CIFAR-100
AUPR
1.00
1.00
0.86
1.00
1.00
0.15 0.98
AUC
1.00
1.00
0.13 1.00
1.00
1.00
0.02 1.00
CelebA
AUPR
1.00
1.00
0.27 0.99
1.00
1.00
0.26 1.00
AUC
1.00
1.00
0.02 0.99
1.00
1.00
0.00
1.00
LSUN
AUPR
1.00
1.00
0.15 0.98
1.00
1.00
0.15 1.00
AUC
0.26
0.68
0.36 0,84
0.14
0.67
0.27 0.94
CIFAR-10 10
AUPR
0.23
0.53
0.27 0.73
0.20
0.52
0.23 0..1
CelebA
AUC
0.29
0.65
0.30 0.82
0.14
0.61
0,20 0.94
CIFAR-100
AUPR
0.25
0.50
0.74
0.21
0.46
0.21 0.90
AUC
1.00
0.83
0.57
1.00
0.88
0.53 1.00
SVHN
AUPR
1.00
0.87
0.67
1.00
0.91
0.65 1.00
AUC
0.29
0.64
0.33 0.85
0.18
0.59
0.22 0.96
LSUN
AUPR
0.25
0.47
0.25 0.75
0.21
0.43
0.22 0.92
Table Comparison of KS and CvM test on RealNVP. AUROC shown (higher is better). Our results are denoted by GODI KODIKS|GOD2A GOD2KS.
batch size =
batch size = 10
batch size =
20
InD/OOD
KS
CvM
KS
CvM
KS
CvM
SPARR10/SVHY
0.86/0.82
0.77
0.99|0.97
0.97
1.00 1.00
1.00
A
0.80
0.82
0.92
0.94
0.99 0.99
1.00
CIFAR-1MLSSUN
0.61
0.63
0.65
0.68
0.72
0.76|0.79
SVHN/CIFAR-10
0.89
0.88
0.95
0.93
0.99 1.00
0.97
or more training time) tends to give more accurate estimation of density. However, the OOD detection is mainly based on both the separation of the InD score and OOD score and their relative order (i.c. InD must have higher likelihood estimation than OOD input, otherwise detection based on the likelihood will fail) instead of their respective accuracy. Some prior works (Schirrmeister et al. 2020 Zhang et al. 2021) reported a related result. Specifically, Schirrmeistrr et al. (2020) modifies the model architecture, that the wrong relative order of InD and OOD density histograms reversed, which leads to the success of OOD detection, although the modified model has worse log likelihood. In contrast, /hang al. (2021) consider successful case, where the relative order of Inl)
19

---

Published as a conference paper at ICLR 2022
Table AUROC and AUPR when batch size 20. Our results are denoted by GODIKSGOD223 We use the boldface for the highest metrics, and underline the detection failure case (where metric below 0.5).
RealNVP, batch size 20
Glow, hatch size
20
InD
OOD
Metric
TyTest
KLOD
Ours
TyTest
KLOD
Ours
AUC
0.83
0.97
1.00
1.00
0.01
0.05 1.00
MNIST
FMNIST
AUPR
0.93
1.00
0.99
0.31
0.31
1.00
AUC
0.98
0.99
1.00
0.58
0.02
0.00
1.00
KMNIST
AUPR
0.97
0.98
1.00
0.60
0.31
0.31
1.00
AUC
1.00
1.00
1.00
1.00
1.00
1.00 1.00
Omniglot
AUPR
1.00
1.00
1.00
L.00
1.00
1.00
AUC
0.99
0.46
1.00
1.00)
1.00
1.00
SVHN
AUPR
1.00
0.78
1.00
1.00
1.00
1.00 1.00
CIFAR-10
AUC
0.91
1.00
0.98
0.78
1.00
0.89 1.00
CelebA
AUPR
0.94
1.00
0.99
0.85
1.00
0.93 1.00
AUC
0.81
0.91
0.73
0.99
0.75
LSUN
AUPR
0.90
0.75 0.80
0.99
0.72
0.53
AUC
1.00
0.41
1.00 1.00
1.00
1.00
1.00 1.00
SVHN
CIFAR-100
AUPR
1.00
0.75
1.00
1.00
1.00
1.00
AUC
0.53
1.00
0.98
0.32
1.00
0.94 1.00
CelebA
AUPR
0.66
1.00
0.99
0.54
1.00
0.97 1.00
AUC
0.60
0.85
0.79
0.87
0.91
0.17 0.63
LSUN
AUPR
0.58
0.81
0.69
0.88
0.91
0.34
AUC
1.00
0.99
0.99) 1.00
1.00
1.00
0.00 1.00
CIFAR-10
AUPR
1.00
0.98
0.94
1.00
1.00
0.15 L.00
SVHN
AUC
1.00
1.00
0.99 1.00
1.00
1.00
0.00 1.00
CIFAR-100
AUPR
1.00
0,99
0.95
1.00
1.00
0.15 1.00
AUC
1.00
1.00
1.00
1.00
1.00
0.00
1.00
CelebA
AUPR
1.00
1.00
1. 1.00
1.00
1.00
0.26 1.00
AUC
1.00
1.00
1.00
1.00
1.00
0.00)
1.00
LSUN
AUPR
1.00
1.00
0.99 1.00
1.00
1.00
0.16 1.00
AUC
1.00
1.00
1.00
0.05
0.67
0. I.00
CIFAR-10 10
AUPR
1.00
1.00
1.00
0.19
0.54
0.21 0.99
CelebA
AUC
1.00
1.00
1.00
0.05
0.57
0.10 0.99
CIFAR-100
AUPR
1.00
1.00
1.00
0.19
0.42
0.20 0.99
AUC
0.85
1.00
1.00
1.00
0.93
0.50 1.00
SVHN
AUPR
1.00
1.00
1.00
0.95
0.62 1.00
AUC
1.00
1.00
1.00
0.10
0.58
0.12 1.00
LSUN
AUPR
1.00
1.00
1.00
0.20
0.44
0.20 1.00
and OOD bpd (bits per dimension) histograms is correct, but they found that a mis-estimated model can lead to a smaller overlap in bpd histograms of InD and OOD, which results the improvement of detection performance (see Figure 2 in Zhang al. (222T). This intuition is also valid in method. Take our one-sa test as an example. For a worse model, the transformed distribution of both InD and OOD in the latent space will be further away from the prior, compared better model, but the separation in this case might be more pronounced, which is why a worse model can sometimes lead better OOD detection.
More importantly, our detection method is not based on likelihood score, and yet we still observe similar phenomenon. These results indeed related, and our results corroborate existing work by examining more flow models, test statistics, real datasets and metrics.
G.7 INTRA-DATASET OOD DETECTION
Till now, all our OOD detection experiments were conducted across different datasets (i.c. inter- dataset OOD). We conduct additional experiments detecting OOD within one dataset, i.e. treating one class as InD, the rest classes combined as OOD. Table Ishow the detection results on FMNIST and CIFAR-10. The batch size is fixed as 10, and our method uses 200 random projections. Com- pared to SOTA baselines, our method achieves the best intra-dataset OOD detection performance on FMNIST dataset for all classes, while CIFAR-10 dataset, our method yields the near-best de-
20

---

Published as a conference paper at ICLR 2022
Figure 8: Model capacity vs. OOD detection. Images are generated from RealNVP trained on CelebA. Assume CIFAR-10 as OOD, and our GODIKS and GOD2KS results are segmented by
I. (a) Generated images from a RealNVP with 1 block and 2048 hidden channels. AUROCs 0.98|0.99. (b) Generated images from RealNVP with 16 blocks and 512 hidden channels. AU-
ROCs = 0.88.
(a)
(b)
(d)
Figure 9: Training epochs vs. OOD detection. Images are generated from RealNVP trained on CelebA. Assume CIFAR-10 OOD, and our GODIKS and GOD2KS results are segmented by (a) Trained for 10 epochs. AUROCs 0. 0.78|0.78. (b) Trained for 100 epochs. AUROCs 0.81. (c) Trained for 150 epochs. AUROCS 0.86)0.86. 86. (d) Trained for 200 epochs. AUROCs 0.88 0.88.
tection performance for all classes. Besides, there is no detection failure case in our method, which serves as another evidence of the effectiveness and generality our proposed method.
G.8 OOD DETECTION ON TIME-SERIES DATA
Till now, all our experiments are conducted on widely used image benchmarks. It will be interesting to investigate whether our method can generalize beyond image datasets. We consider testing phys- iological signals semees data), e.g. Electrocardiography (ECG) and Electroencephallggaphh (EEG). Clinically, ECG and EEG are electric signals that could reflect heart or brain activities, thus be critical for the monitoring and diagnosis of cardiac dysfunctions neural disorders. In this experiment, we train RealNVP on one signal, and test the OOD detection performance by treating the other kind of signal OOD. Our experimental data come from publicly available database. i.c. MIT-BIH Polysomnographic database (Ichimaru & Moody 1999 Goldberger et al. 2000) The MIT-BIH Polysomnographic Database collects various physiological signals, including EEG and ECG, from 16 subjects during sleep. We randomly select four records from the database for
analysis, i.e. slp03, slp04, slp14, slp16, where each record is exactly 6-hour long. Amplitudes are
normalized between and 1. The whole signal is segmented into 10-second strips, so the size of
21

---

Published as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
Figure 10: Training epochs vs. OOD detection. Images are generated from a Glow with K
16, L = 3, 128 trained on CIFAR-10. Treating SVHN as OOD, our GODIKS and GOD2KS results are segmented by |. (a) Trained for epochs. AUROCs 0.96|0.96, (b) Trained for 10
epochs. AUROCS 0.96. (c) Trained for 100 epochs. AUROCs 0. 0.76/0.84. (d) Trained for 200 epochs. AUROCs = 0. 0.84
Ai
(a)
(b)
(d)
Figure 11: Training epochs vs. OOD detection. Images generated from Glow with K
16, L = 2, h = 128 trained on FMNIST. Treating MNIST as OOD, our GODIKS and GOD2KS results are segmented by J. (a) Trained for 5 epochs. AUROCs 1.00|1.00. (b) Trained for 10 epochs. AUROCS 1.00]1.00. (c) Trained for 100 epochs. AUROCs = 0.90]1.00. (d) Trained for 200 epochs. AUROCS 0.92|1.00.
this toy dataset is 8640, where 10% is randomly held out for validation, and another 10% is ran- domly held out for test. The rest 80% is used for training. The sampling rate 250 Hz, so the data dimension is 2500. Table 10 shows the detection performance of OOD detection and EEG/ECG by our method as well as two SOTA baselines, where the batch size is fixed as 10 and we 200 random projections for our methods. It can be observed from Table 10 that our method, especially GOD2KS, indicates more robustness against SOTA baselines. In the EEG/ECG setting, typicality test fails, implying an overlap density histograms of InD and OOD, and KLOD is slightly better than random guess. In contrast, our GOD2KS achieves around 0,9 metrics in both experimental settings.
ECG/EEG
G.9 INVESTIGATION OF CIFARS VS. LSUN
For CIFAR vs LSUN, our experimental results indicate that this is challenging case for our method. We examined the raw distributions of the two datasets, through per-dimensional mean and variance. The detailed result is given in Figure 12 where we verify that CIFARs and LSUN share a very close per-dimensional mean and variance, while for CIFARs vs. SVHN, although their per-dimensional mean share a similar range, their distributional shapes are different in terms of per-dimensional variances. This implies the CIFARS vs. LSUN pair is distributionally more similar and hence more challenging for our methods. The immediate solution is to increase the batch size. When batch size 100, our methods (using RealNVP) and 0.90 for CIFAR-100 vs. LSUN. When batch size is beyond 200, our methods can achieve perfect detection performance in both settings.
achieve AUROC of 0.97|1.00 for CIFAR-10 10 vs. LSUN,
22

---

Published as a conference paper at ICLR 2022
Table 9: Intra-dataset OOD detection using RealNVP. AUROC and AUPR are shown (higher is better). The batch size is fixed as 10. Our results are denoted by GODIKS/GOD2KS.
FMNIST
CIFAR-10
Classes
Metrics
TyTest
KLOD
Ours
TyTest
KLOD
Ours
AUC
0.99
1.00
1.00
0.93
0.98
0.94[0.97
0
AUPR
1.00
1.00
1.00] 1.00
1.00
0.99
1.00
0.99|0.99
AUC
1.00
1.00
1.00
0.50
0.60
0.69/0.68
1
AUPR
1.00
1.00
1.00
0.89
0.93
0.95
AUC
0.96
0.97
0.99 0.99
0.88
0.83
0.75
2
AUPR
0.99
1.00
1.00 1.00
0.98
0.97
0.95
AUC
1.00
1.00
1.00) 1.00
0.61
0.68
0.63
3
AUPR
1.00
1.00
1.00) 1.00
0.93
0.94
0.93
AUC
0.98
0.98
1.00 1.00
0.98
0.97
0.93
4
AUPR
1.00
1.00
1.00) 1.00
1.00
1.00
0.99|0.99
AUC
1.00
1.00
1.00) 1.00
0.68
0.80
0.75
5
AUPR
1.00
1.00
1.00 1.00
0.95
0.97
0.96/0.96
AUC
0.88
0.90
0.95 0.96
0.89
0.93
0.91
6
AUPR
0.98
0.99
0.99 1.00
0.99
0.99
0.99
AUC
1.00
1.00
1.00 1.00
0.58
0.64
0.66
7
AUPR
1.00
1.00
0.93
0.94
0.94
AUC
0.74
0.86
0.99 0.99
0.99
0.97
0.96
AUPR
0.94
0.98
1.00
1.00
1.00
1.00
AUC
1.00
1.00
1.00 1.00
0.51
0.81
0.82 0.81
9
AUPR
1.00
1.00
1.00
0.90
0.98
0.98 0.99
Table 10: OOD detection on EEG and ECG using RealNVP. AUROC and AUPR are shown (higher is better). The batch size is fixed 10. Our results are denoted by GODIKS
TyTest
KLOD
Ours
InD/OOD
AUC
AUPR
AUC
AUPR
AUC
AUPR
EEG/ECG
0.15
0.33
0.60
0.50
0.50/0.92
0.47|0.88
ECG/EEG
1.00
1.00
0.99
0.99
0.90/0.92
0.92[0.94
G.10 COMPARISON WITH AVERAGE DENSITY
For thoroughness we also include comparison against a straightforward baseline, i.e. using the averaged density across a batch as the detection score (we denote this approach as Avg in Table We conduct additional experiments on FMNIST/MNIST. CIFAR-10/SVHN (the two most popular experimental settings in OOD detection) using both RealNVP and Glow, as shown Table [T The batch size is fixed 10. The results of other methods are directly cited from Table and can be observed that directly averaging the density across a batch is not competitive. The result is surprising, because long the density histograms of Inl and OOD sit in the wrong relative position, the average density will not work, regardless of batch size.
and Table It
23

---

Published as a conference paper at ICLR 2022
0.31
600
4:50
0.50
0.0
130
330-
1800
ABO Ed1 COR B30 CUN Kik Eim
540 3431 MGS ARN ROR 953 021
6300 0300 0:08 600 BS0 KDI Xicc 32%
180 1201 136 029 CDN
(a) CIFAR-10 vs. LSUN (b) CIFAR-100 100 vs. LSUN (c) CIFAR-10 10 vs. SVHN (d) CIFAR-100 vs. SVHN
Figure 12: Per-dimensional mean and variance comparison. (a)-(b) Per-dimensional mean and vari- ance of CIFARs and LSUN are close, which implies that CIFARs and LSUN share similar distri- bution of normalized pixel values. (c)-(d) As comparison, CIFARs vs SVHN have more clearly separable per-dimensional variance.
Table 11: Comparison with averaged density. Our results are denoted by (ODIKS/GGD2KS We use the boldface for the highest metric, and underline the detection failure case (where metric is below 0.5).
RealNVP, batch size
Glow, batch size
InD/OOD
Metrics
Avg
TyTest
KLOD
Ours
Avg
TyTest
KLOD
Ours
AUC
0.83
0.82
0.96
1.00
0.00
1.00
0.04
0.11
FMNISTMMIIST
AUPR
0.69
0.68
0.89
1.00
0.31
1.00
0.31
1.00
AUC
0.01
0.95
0.35
0.99
0.00
1.00
1.00
1.00
CIFAR-IO/SVHN
AUPR
0.51
0.98
0.72
1.00
0.51
1.00
1.00
1.00
24


Reviews:
[{"review_id": "wwGhz24elvD", "reviewer": "Reviewer_zvSD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n*Exploration of OOD in practice*:  While GOF-based OOD detection is theoretically well-founded, applying the technique successfully in practice for high-dimensional data is a challenge in its own right.  Thus, this paper provides value to practitioners, guiding them on how to make these crucial implementation choices.\n\n*Ablations*: The paper also does a nice job of testing the efficacy of specific parts of the experimental pipeline---for example, autoencoders vs random projections, GOF vs two-sample tests.\n\n## Cons\n\n*Novel only in implementation*:  The method's novelty is restricted to the implementation details.  Ultimately, the core methodology is to apply the KS test to a reparameterized distribution, which is not an original contribution.  Applying a KS test to flows for OOD purposes is novel, to the best of my knowledge.  Moreover, the paper's other insights such as model quality vs OOD ability mostly back already supported points from earlier work (e.g. [Zhang et al, ICML 2021].", "summary_of_the_paper": "This paper applies a goodness of fit test (KS test) to the latent space of normalizing flows for purposes of out-of-distribution detection.  To combat high-dimensionality and model misspecification, extensions such as random projections and two-sample tests are also proposed.  The results report AUROC on RNVP (batch size 5 and 10), comparing against a typicality test and a kl-based test.  The paper also tests using autoencoders vs rand projections, alternative divergences, effects of model misspecification, and detection in latent vs original feature space.  The findings are that the KS-test is indeed a practical choice for OOD detection via GOF testing.  Moreover, it is observed that better models don't necessarily mean better OOD detection, as described earlier by Zhang et al. [ICML 2021].", "summary_of_the_review": "The paper is fairly well executed; however, I can't say that I learned much from the work as its contribution is mostly in the implementation details.  And it is unclear to what extent these details generalize beyond standard image benchmarks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sje3NF3KoO", "reviewer": "Reviewer_R73v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nS1. The proposed approach is interesting, simple, mathematically sound and effective. In comparison with Zhang et al. (2020), the proposed approach does not require estimation of a multivariate distribution.\n\nS2. Experiments show robust overall performance. Notably, there are less failure cases than in related methods.\n\nWeaknesses:\n\nW1: Authors does not propose a compelling motivation for groupwise OOD detection.\n\nW2: The evaluation should also show the OOD AP metric as in Zhang et al. (2020). \n\nSuggestions\n- Can you offer an intuitive explanation for the failures of TyTest and KLOD in Table 1?\n- It may be interesting to evaluate the average density across the input batch as a straight-forward baseline. I do not expect this to be competitive, but averaging may improve over the pointwise result.\n- Can you provide any insight why the method can not exploit normalizing flows with large capacity?\n- Include algorithmic presentation of GOD2KS in the supplement (for completeness).\n- Page 2: absolute value of the Jacobian -> absolute value of the determinant of the Jacobian.\n- Figure 1, caption: explain S_0 and S_1.\n- Page 3, \"in order to evade the curse of dimensionality\": this becomes especially relevant in contemporary normalizing flows (eg. VFlow, DenseFlow) where latent dimensionality is greater than input dimensionality.\n- Page 3, \"is distinct from the distributions\": make clear that this refers only to Fig1(b).\n- Figure 2, caption: does not follow -> follow.\n", "summary_of_the_paper": "The manuscript addresses groupwise outlier detection with normalizing flows. The main idea is to project latent representations of the input batch onto several random directions and to evaluate the resulting 1D distributions according to the Kolmogorov-Smirnov test either with respect to the prior (GOD1KS) or with respect to the corresponding distribution of the training data (GOD2KS). Finally, OOD ranking is performed according to the average KS value across all random directions. GOD1KS is used when the underlying normalizing flow is well learned, while GOD2KS is an option when there is some uncertainty regarding the learning outcome. Experimental performance decreases  when the method is used in combination with undertrained normalizing flow or a normalizing flow with a larger capacity.", "summary_of_the_review": "This paper provides interesting insights about normalizing flows and OOD detection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lvELoStMHKI", "reviewer": "Reviewer_Kij3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe proposed approach is simple and does not require training additional components. It can be applied both in data space and latent of the flow, but according to experimental studies application of latent space gives better results. The results are comparable to the results obtained to the reference approaches - the significant gain is observed on CIFAR10 vs SVHN setting. The paper is easy to follow and the idea of the paper is clear. The proposed approach is easy to applied so the results are easy to reproduce.\n\nWeaknesses \n\nThe contribution of this work is smart but novelty is a bit limited due to the fact that the authors combine two known approaches - KS statistical test and random projections. It seems that this framework is general and can be even applied to the bottleneck models like VAE or even for any low dimensional features extracted from the data. The question is can we apply this framework to any low dimensional representation of data or we need the invertible mapping that does not decrease the dimension and direct access to the likelihood. \n\nThe second question is about Carleman’s condition. Considering the case where we use flow’s latent is it sufficient that it would be satisfied by Gaussian (flow latent distribution) or it should be satisfied by data distribution (the normalised Gaussian by determinant of the Jacobian)?\n\nThe third issue is about the results. The model seems to be comparable to KLOD besides the CIFAR10 vs SVHN where it performs better. On the other hand, it has problems with CIFAR vs LeSUN setting. Do you have some intuition what is the reason and how to deal with that issue? Most of the experiments that examine the OOD problem are performed between different datasets. It would be beneficial to see some analysis for inside dataset split, where the split is indicated by two groups of classes. This case is somehow similar to Cifar10 vs. Cifar100 comparison. ", "summary_of_the_paper": "The paper provides an interesting approach for OOD detection problem for flow models. The approach is based on random projections on the real line where KD statistical test is applied in order to compare two distributions. Two variants of the approach are considered 1 sample test where the comparison is made with respect to flow prior and 2 sample test where the comparison is made with respect to transformed samples from two datasets. The quality of the approach is compared agains standard reference methods. \n", "summary_of_the_review": "I admire the simplicity of the approach and I tent to recommend to accept this work at this stage. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bba6HqjCSm", "reviewer": "Reviewer_XXZp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found it easy to understand the ideas and methods of the paper, results are also well-understandable.\n\nOverall, the paper does not bring too many surprises from my view - I would have expected any use of 1d projections for OOD detection to work better in latent space than in input space. Also it seems not too surprising that this very generic method cannot result in very high OOD detection performance (compared to methods exploiting more assumptions about OOD data). But in any case good to see precisely how well OOD detection works in this way. The main surprising part might be that generation quality can be negatively correlated with OOD detection performance.\n\nThe manuscript would benefit from discussing other OOD detection approaches in latent space, https://arxiv.org/abs/2006.10848 also present one method using parts of the latent space of the Glow model, https://arxiv.org/abs/2102.08248 uses VAEs, not flows, but also look at parts of the latent space there. In the future, could be interesting to compare to such methods (using parts of latent space/log-likelihood contributions), which exploit more assumptions about where semantic differences are encoded in the latent space, on both semantic and non-semantic OOD detection. Note also that https://arxiv.org/abs/2006.10848 report a related result to the result that generation quality and OOD detection performance can be negatively correlated, finding that fully connected flow models have worse log likelihoods but better OOD detection performance (see Sec 2).\n\nI would like it if caption figures were more self-contained and figures could be understood separately from the text. E.g., figure 3 and figure 5 captions could also talk a bit more about what one could learn from the figures.", "summary_of_the_paper": "The paper proposes evaluates using statistical tests on random 1d-projections in the latent space of a flow model for groupwise out-of-distribution (OOD) detection. Concretely, Komolgorov-Smirnov tests are used to compare latent encodings of a given batch of samples to the expected distribution of the in-distribution latent encodings. For the expected distribution, either the predefined latent prior is used, resulting in a 1-sample Komolgorov-Smirnov test, or the empirical distribution of the latent encodings of the training data  is used, resulting in a 2-sample Komolgorov-Smirnov test. The paper reports similar groupwise OOD detection performance as existing methods, with the 2-sample KS-test method having less failure cases.", "summary_of_the_review": "The paper in my view does not present too many surprising results, but I also find it valuable to see non-surprising results that are clearly presented and easy to understand. And the result on generation quality and OOD detection performance being negatively correlated is also valuable to know.\n\n**Update Post Rebuttal**\nThe revisions have further improved the manuscript, I increase my score to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the use of flow models for out-of-distribution detection. The paper proposes to use a combination of random projections in the latent space of flow models and one-sample / two-sample statistical tests for detecting OOD inputs. The authors present results on image benchmarks as well as non-image benchmarks. 

The reviewers found the approach well-motivated and appreciated the ablations. The authors did a good job of addressing reviewer concerns during the rebuttal. During the discussion phase, the consensus decision leaned towards acceptance. I recommend accept and encourage the reviewers to address any remaining concerns in the final version.

It might be worth discussing this paper in the related work: Density of States Estimation for Out-of-Distribution Detection https://arxiv.org/abs/2006.09273
Decision: Accept (Poster)