Revisiting flow generative models for Out-of-distribution detection

Deep generative models have been widely used in practical applications such as the detection of out-of-distribution ( OOD ) data . In this work , we aim to re-examine the potential of generative flow models in OOD detection . We first propose a simple combination of univariate one-sample statistical test ( e.g. , Kolmogorov-Smirnov ) and random projections in the latent space of flow models to perform OOD detection . Then , we propose a two-sample version of our test to account for imperfect flow models . Quite distinctly , our method does not pose parametric assumptions on OOD data and is capable of exploiting any flow model . Experimentally , firstly we confirm the efficacy of our method against state-of-theart baselines through extensive experiments on several image datasets ; secondly we investigate the relationship between model accuracy ( e.g. , the generation quality ) and the OOD detection performance , and found surprisingly that they are not always positively correlated ; and thirdly we show that detection in the latent space of flow models generally outperforms detection in the sample space across various OOD datasets , hence highlighting the benefits of training a flow model . 1 INTRODUCTION . Reliably detecting out-of-distribution ( OOD ) data , also known as anomaly , plays an important role in many deployed machine learning systems , particularly so when the system may be under attack by a malicious adversary ( Markou & Singh , 2003a ; b ; Toth & Chawla , 2018 ; Chalapathy & Chawla , 2019 ) . The bulk of existing OOD detection algorithms , in one way or another , boils down to thresholding the ( density ) likelihood , which can be conveniently estimated by modern flow-based generative models . However , quite surprisingly , recent studies ( Nalisnick et al. , 2019a ) reveal that flows that are trained to maximize the likelihood of in-distribution ( InD ) data may actually assign a higher likelihood to OOD data . For example , the flow-based model Glow ( Kingma & Dhariwal , 2018 ) trained on CIFAR-10 that contains natural images ( e.g. , dog , cat , and ship ) is found to assign a higher likelihood to SVHN that consists of house numbers . Such surprising and counter-intuitive observations lead to natural questions on the applicability of flow models for performing the OOD detection by merely thresholding the log-likelihood . In this work , we re-examine the ability of flow-based generative models for OOD detection . Flow models typically transform a prior distribution that is easy to sample from to the data distribution via an invertible mapping ( Tabak & Vanden-Eijnden , 2010 ; Rezende & Mohamed , 2015 ; Dinh et al. , 2017 ; Kingma & Dhariwal , 2018 ) . Building on this property , we propose to compare the training and test samples in the latent space of a flow model , as opposed to relying on point-wise scoring functions such as the log-likelihood . To this end , we divide test samples into minibatches ( groups ) . Examples of group OOD detection with generative models include for instance Nalisnick et al . ( 2019b ) ; Song et al . ( 2019 ) ; Chalapathy et al . ( 2018 ) ; Zhang et al . ( 2020 ) , most of which consider either the raw input or certain representation of the raw input in the sample space for detection . In contrast , we exploit data representations in the latent space , and confirm its advantage experimentally . Moreover , to cope with the curse of dimensionality ( as is typical in image datasets ) , we propose to leverage random projections ( Friedman et al. , 1984 ) , which frees one from designing any extra network architectures and is computationally very efficient . Our proposed detection algorithm combines classic univariate statistical tests and random projections in the latent space of flow models . Besides the scoring function used for detection and the representation of data , model accuracy also greatly affects OOD detection performance ( Zhang et al. , 2021 ; Choi et al. , 2018 ) . To account for imperfectly trained flow models , we propose a two-sample version of our detection algorithm for practical use . Surprisingly , in our experiments , we confirm that model accuracy , indicated by its generation quality , may not always be positively correlated with its OOD detection performance . We summarize our contributions as follows : • We propose the OOD detection algorithms GOD1KS and GOD2KS for ideal and imperfect flow models , respectively , which pose no parametric assumption on the OOD data . To evade the curse of dimensionality , we propose to randomly project the latent space of flow models onto the real line and perform univariate statistical tests ( such as Kolmogorov-Smirnov , KS ) there . Our method is computationally very efficient , requires no extra network architecture , and unifies OOD detection in both sample space and latent space . • Experimentally , 1 ) we compare with the state-of-the-art benchmarks on various image datasets and obtain competitive results ; 2 ) we confirm on larger models and real datasets that model accuracy may not always be positively correlated with OOD detection performance ; and 3 ) we compare detection in the sample space versus in the latent space of the flow model , and reveal the superiority and robustness of the latter , hence highlighting the potential of flow models in OOD detection . 2 OOD DETECTION WITH FLOW MODELS . In this section , we state the OOD detection problem and propose statistical tests that exploit modern flow generative models ( either perfect or imperfectly trained ) . 2.1 OOD DETECTION : GROUP VS. POINTWISE . Let D = { X1 , . . . , Xn } i.i.d.∼ p be a sample from an in-distribution ( InD ) p. Our goal is to construct a statistical test that can decide if a test sample { Y1 , . . . , Ym } is from p ( InD ) or some unknown out-of-distribution ( OOD ) q . When m = 1 , i.e . we examine one test sample at a time , it is often called the pointwise OOD detection ( POD ) while group OOD detection ( GOD ) refers to m > 1 . According to Xiong et al . ( 2011 ) , group OOD can be characterized as ( 1 ) point based , where each individual point in the group is anomalous , and ( 2 ) distribution based , where a group of points shows a different pattern while any single point may seem regular . Therefore , distribution based group anomaly is only detectable by GOD methods . Practical applications include detecting Higgs Bosons as a group of collision events in high-energy particle physics ( Muandet & Schölkopf , 2013 ) , and detecting distributed denial-of-services attacks via a group of multi-sensor networks ( Chen & Yu , 2016 ) . We construct a synthetic Gaussian experiment in appendix E to illustrate this point . Thus in this work , we will focus on studying the applicability of modern flow generative models to the group OOD detection problem . 2.2 FLOW-BASED GENERATIVE MODELS . A flow-based generative model simply learns a transformation T , typically a diffeomorphism , that pushes a latent distribution p0 ( e.g. , Gaussian or uniform ) to the in-distribution p , i.e . Z ∼ p0 =⇒ T ( Z ) ≈ p , also denoted as p ≈ T # p0 . ( 1 ) In particular , when T is diffeomorphic , we have the familiar change-of-variable formula : pT ( x ) = p0 ( z ) /|T′ ( z ) | = p0 ( T−1 ( x ) ) / ∣∣T′ ( T−1 ( x ) ) ∣∣ , ( 2 ) where |T′ ( z ) | denotes the absolute value of the determinant of the Jacobian of T. A flow model parameterizes T through a neural network and estimates T by minimizing some divergence between pT ( the model density ) and p ( the data density ) , based on the training data D. Once T is learned , we can easily generate new data by simply sampling Z from p0 and then pushing through T , i.e . X = T ( Z ) . Denote X1 d = X2 if they follow the same distribution . For any transformation T , it is clear that X1 d = X2 =⇒ T ( X1 ) d = T ( X2 ) , ( 3 ) and the converse also holds if T is invertible . In fact , a sufficiently regular transformation T may preserve many familiar statistical divergences D , i.e . D ( X1 , X2 ) = D ( T ( X1 ) , T ( X2 ) ) , ( 4 ) for instance , when D is the f -divergence , such as the Kullback-Leibler ( KL ) and Jensen-Shannon ( JS ) divergence , T is diffeomorphic , and X1 and X2 are continuous ( Csiszár , 1963 ) . Another example is the Kolmogorov-Smirnov ( KS ) distance for real-valued Xi and monotonic T : D ( X1 , X2 ) : = supx |F1 ( x ) − F2 ( x ) | , ( 5 ) where Fi is the CDF of Xi , as well as the Cramér-von Mises ( CvM ) divergence ( Darling , 1957 ) : D ( X1 , X2 ) : = ∫ [ F1 ( x ) − F2 ( x ) ] 2dF2 ( x ) . ( 6 ) Thus , for group OOD detection , we can first train an invertible flow model based on the InD samples and then construct a scoring function or statistical test either in the sample space ( where the test sample { Yj } resides ) , or in the latent space ( where { T−1 ( Yj ) } resides ) . However , since both Yj and its pre-image T−1 ( Yj ) are typically of high dimension and the sample sizem is comparatively small , some compromise needs to be made in order to evade the curse of dimensionality , which gets even more problematic for flow models whose latent dimension is greater than input dimension ( Chen et al. , 2020a ; Grcić et al. , 2021 ) . For instance , Zhang et al . ( 2020 ) fit a multivariate Gaussian in the latent space using { T−1 ( Yj ) } and compute analytically the KL divergence between two Gaussians . Nalisnick et al . ( 2019b ) project Yi onto the real line using the estimated log-likelihood function log pT and construct a typicality test there , while Choi et al . ( 2018 ) instead employ the WatanabeAkaike Information Criterion ( WAIC ) . To illustrate the above idea , we compare in Figure 1 the distributions of T−1 ( Xtrain ) and T−1 ( Xtest ) in the latent space using synthetic 2-D Gaussian datasets and the flow model RealNVP ( Dinh et al. , 2017 ) . We observe that in the latent space the distribution of the OOD samples ( i.e. , T−1 ( XOOD-test ) ) is distinct ( more so in Fig 1 ( b ) than Fig 1 ( a ) ) from the distribution of the InD samples ( i.e. , both T−1 ( XInD-train ) and T−1 ( XInD-test ) ) . Moreover , we found that the transformation T−1 can roughly keep the statistical distance in the latent space . In other words , distributions that are far away from the in-distribution in the sample space also tend to remain far away in the latent space .

Summary: The paper proposes evaluates using statistical tests on random 1d-projections in the latent space of a flow model for groupwise out-of-distribution (OOD) detection. Concretely, Komolgorov-Smirnov tests are used to compare latent encodings of a given batch of samples to the expected distribution of the in-distribution latent encodings. For the expected distribution, either the predefined latent prior is used, resulting in a 1-sample Komolgorov-Smirnov test, or the empirical distribution of the latent encodings of the training data  is used, resulting in a 2-sample Komolgorov-Smirnov test. The paper reports similar groupwise OOD detection performance as existing methods, with the 2-sample KS-test method having less failure cases.