Revisiting flow generative models for Out-of-distribution detection

Deep generative models have been widely used in practical applications such as the detection of out-of-distribution ( OOD ) data . In this work , we aim to re-examine the potential of generative flow models in OOD detection . We first propose a simple combination of univariate one-sample statistical test ( e.g. , Kolmogorov-Smirnov ) and random projections in the latent space of flow models to perform OOD detection . Then , we propose a two-sample version of our test to account for imperfect flow models . Quite distinctly , our method does not pose parametric assumptions on OOD data and is capable of exploiting any flow model . Experimentally , firstly we confirm the efficacy of our method against state-of-theart baselines through extensive experiments on several image datasets ; secondly we investigate the relationship between model accuracy ( e.g. , the generation quality ) and the OOD detection performance , and found surprisingly that they are not always positively correlated ; and thirdly we show that detection in the latent space of flow models generally outperforms detection in the sample space across various OOD datasets , hence highlighting the benefits of training a flow model . 1 INTRODUCTION . Reliably detecting out-of-distribution ( OOD ) data , also known as anomaly , plays an important role in many deployed machine learning systems , particularly so when the system may be under attack by a malicious adversary ( Markou & Singh , 2003a ; b ; Toth & Chawla , 2018 ; Chalapathy & Chawla , 2019 ) . The bulk of existing OOD detection algorithms , in one way or another , boils down to thresholding the ( density ) likelihood , which can be conveniently estimated by modern flow-based generative models . However , quite surprisingly , recent studies ( Nalisnick et al. , 2019a ) reveal that flows that are trained to maximize the likelihood of in-distribution ( InD ) data may actually assign a higher likelihood to OOD data . For example , the flow-based model Glow ( Kingma & Dhariwal , 2018 ) trained on CIFAR-10 that contains natural images ( e.g. , dog , cat , and ship ) is found to assign a higher likelihood to SVHN that consists of house numbers . Such surprising and counter-intuitive observations lead to natural questions on the applicability of flow models for performing the OOD detection by merely thresholding the log-likelihood . In this work , we re-examine the ability of flow-based generative models for OOD detection . Flow models typically transform a prior distribution that is easy to sample from to the data distribution via an invertible mapping ( Tabak & Vanden-Eijnden , 2010 ; Rezende & Mohamed , 2015 ; Dinh et al. , 2017 ; Kingma & Dhariwal , 2018 ) . Building on this property , we propose to compare the training and test samples in the latent space of a flow model , as opposed to relying on point-wise scoring functions such as the log-likelihood . To this end , we divide test samples into minibatches ( groups ) . Examples of group OOD detection with generative models include for instance Nalisnick et al . ( 2019b ) ; Song et al . ( 2019 ) ; Chalapathy et al . ( 2018 ) ; Zhang et al . ( 2020 ) , most of which consider either the raw input or certain representation of the raw input in the sample space for detection . In contrast , we exploit data representations in the latent space , and confirm its advantage experimentally . Moreover , to cope with the curse of dimensionality ( as is typical in image datasets ) , we propose to leverage random projections ( Friedman et al. , 1984 ) , which frees one from designing any extra network architectures and is computationally very efficient . Our proposed detection algorithm combines classic univariate statistical tests and random projections in the latent space of flow models . Besides the scoring function used for detection and the representation of data , model accuracy also greatly affects OOD detection performance ( Zhang et al. , 2021 ; Choi et al. , 2018 ) . To account for imperfectly trained flow models , we propose a two-sample version of our detection algorithm for practical use . Surprisingly , in our experiments , we confirm that model accuracy , indicated by its generation quality , may not always be positively correlated with its OOD detection performance . We summarize our contributions as follows : • We propose the OOD detection algorithms GOD1KS and GOD2KS for ideal and imperfect flow models , respectively , which pose no parametric assumption on the OOD data . To evade the curse of dimensionality , we propose to randomly project the latent space of flow models onto the real line and perform univariate statistical tests ( such as Kolmogorov-Smirnov , KS ) there . Our method is computationally very efficient , requires no extra network architecture , and unifies OOD detection in both sample space and latent space . • Experimentally , 1 ) we compare with the state-of-the-art benchmarks on various image datasets and obtain competitive results ; 2 ) we confirm on larger models and real datasets that model accuracy may not always be positively correlated with OOD detection performance ; and 3 ) we compare detection in the sample space versus in the latent space of the flow model , and reveal the superiority and robustness of the latter , hence highlighting the potential of flow models in OOD detection . 2 OOD DETECTION WITH FLOW MODELS . In this section , we state the OOD detection problem and propose statistical tests that exploit modern flow generative models ( either perfect or imperfectly trained ) . 2.1 OOD DETECTION : GROUP VS. POINTWISE . Let D = { X1 , . . . , Xn } i.i.d.∼ p be a sample from an in-distribution ( InD ) p. Our goal is to construct a statistical test that can decide if a test sample { Y1 , . . . , Ym } is from p ( InD ) or some unknown out-of-distribution ( OOD ) q . When m = 1 , i.e . we examine one test sample at a time , it is often called the pointwise OOD detection ( POD ) while group OOD detection ( GOD ) refers to m > 1 . According to Xiong et al . ( 2011 ) , group OOD can be characterized as ( 1 ) point based , where each individual point in the group is anomalous , and ( 2 ) distribution based , where a group of points shows a different pattern while any single point may seem regular . Therefore , distribution based group anomaly is only detectable by GOD methods . Practical applications include detecting Higgs Bosons as a group of collision events in high-energy particle physics ( Muandet & Schölkopf , 2013 ) , and detecting distributed denial-of-services attacks via a group of multi-sensor networks ( Chen & Yu , 2016 ) . We construct a synthetic Gaussian experiment in appendix E to illustrate this point . Thus in this work , we will focus on studying the applicability of modern flow generative models to the group OOD detection problem . 2.2 FLOW-BASED GENERATIVE MODELS . A flow-based generative model simply learns a transformation T , typically a diffeomorphism , that pushes a latent distribution p0 ( e.g. , Gaussian or uniform ) to the in-distribution p , i.e . Z ∼ p0 =⇒ T ( Z ) ≈ p , also denoted as p ≈ T # p0 . ( 1 ) In particular , when T is diffeomorphic , we have the familiar change-of-variable formula : pT ( x ) = p0 ( z ) /|T′ ( z ) | = p0 ( T−1 ( x ) ) / ∣∣T′ ( T−1 ( x ) ) ∣∣ , ( 2 ) where |T′ ( z ) | denotes the absolute value of the determinant of the Jacobian of T. A flow model parameterizes T through a neural network and estimates T by minimizing some divergence between pT ( the model density ) and p ( the data density ) , based on the training data D. Once T is learned , we can easily generate new data by simply sampling Z from p0 and then pushing through T , i.e . X = T ( Z ) . Denote X1 d = X2 if they follow the same distribution . For any transformation T , it is clear that X1 d = X2 =⇒ T ( X1 ) d = T ( X2 ) , ( 3 ) and the converse also holds if T is invertible . In fact , a sufficiently regular transformation T may preserve many familiar statistical divergences D , i.e . D ( X1 , X2 ) = D ( T ( X1 ) , T ( X2 ) ) , ( 4 ) for instance , when D is the f -divergence , such as the Kullback-Leibler ( KL ) and Jensen-Shannon ( JS ) divergence , T is diffeomorphic , and X1 and X2 are continuous ( Csiszár , 1963 ) . Another example is the Kolmogorov-Smirnov ( KS ) distance for real-valued Xi and monotonic T : D ( X1 , X2 ) : = supx |F1 ( x ) − F2 ( x ) | , ( 5 ) where Fi is the CDF of Xi , as well as the Cramér-von Mises ( CvM ) divergence ( Darling , 1957 ) : D ( X1 , X2 ) : = ∫ [ F1 ( x ) − F2 ( x ) ] 2dF2 ( x ) . ( 6 ) Thus , for group OOD detection , we can first train an invertible flow model based on the InD samples and then construct a scoring function or statistical test either in the sample space ( where the test sample { Yj } resides ) , or in the latent space ( where { T−1 ( Yj ) } resides ) . However , since both Yj and its pre-image T−1 ( Yj ) are typically of high dimension and the sample sizem is comparatively small , some compromise needs to be made in order to evade the curse of dimensionality , which gets even more problematic for flow models whose latent dimension is greater than input dimension ( Chen et al. , 2020a ; Grcić et al. , 2021 ) . For instance , Zhang et al . ( 2020 ) fit a multivariate Gaussian in the latent space using { T−1 ( Yj ) } and compute analytically the KL divergence between two Gaussians . Nalisnick et al . ( 2019b ) project Yi onto the real line using the estimated log-likelihood function log pT and construct a typicality test there , while Choi et al . ( 2018 ) instead employ the WatanabeAkaike Information Criterion ( WAIC ) . To illustrate the above idea , we compare in Figure 1 the distributions of T−1 ( Xtrain ) and T−1 ( Xtest ) in the latent space using synthetic 2-D Gaussian datasets and the flow model RealNVP ( Dinh et al. , 2017 ) . We observe that in the latent space the distribution of the OOD samples ( i.e. , T−1 ( XOOD-test ) ) is distinct ( more so in Fig 1 ( b ) than Fig 1 ( a ) ) from the distribution of the InD samples ( i.e. , both T−1 ( XInD-train ) and T−1 ( XInD-test ) ) . Moreover , we found that the transformation T−1 can roughly keep the statistical distance in the latent space . In other words , distributions that are far away from the in-distribution in the sample space also tend to remain far away in the latent space .

Summary: This paper applies a goodness of fit test (KS test) to the latent space of normalizing flows for purposes of out-of-distribution detection.  To combat high-dimensionality and model misspecification, extensions such as random projections and two-sample tests are also proposed.  The results report AUROC on RNVP (batch size 5 and 10), comparing against a typicality test and a kl-based test.  The paper also tests using autoencoders vs rand projections, alternative divergences, effects of model misspecification, and detection in latent vs original feature space.  The findings are that the KS-test is indeed a practical choice for OOD detection via GOF testing.  Moreover, it is observed that better models don't necessarily mean better OOD detection, as described earlier by Zhang et al. [ICML 2021].