Exploring the Correlation between Likelihood of Flow-based Generative Models and Image Semantics

Among deep generative models , flow-based models , simply referred as flows in this paper , differ from other models in that they provide tractable likelihood . Besides being an evaluation metric of synthesized data , flows are supposed to be robust against out-of-distribution ( OoD ) inputs since they do not discard any information of the inputs . However , it has been observed that flows trained on FashionMNIST assign higher likelihoods to OoD samples from MNIST . This counterintuitive observation raises the concern about the robustness of flows ’ likelihood . In this paper , we explore the correlation between flows ’ likelihood and image semantics . We choose two typical flows as the target models : Glow , based on coupling transformations , and pixelCNN , based on autoregressive transformations . Our experiments reveal surprisingly weak correlation between flows ’ likelihoods and image semantics : the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged , which we call semantic-invariant transformations ( SITs ) . We explore three SITs ( all small pixel-level modifications ) : image pixel translation , random noise perturbation , latent factors zeroing ( limited to flows using multi-scale architecture , e.g . Glow ) . These findings , though counter-intuitive , resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels . So flows ’ likelihoods , modeling on pixel-level intensities , is not able to indicate the existence likelihood of the high-level image semantics . We call for attention that it may be abuse if we use the predictive likelihoods of flows for OoD samples detection . 1 INTRODUCTION . Deep generative models have been very successful in image generation ( Brock et al. , 2018 ; Kingma & Dhariwal , 2018 ; Miyato et al. , 2018 ) , natural language generation ( Bowman et al. , 2015 ; Yu et al. , 2017 ) , audio synthesis ( Van Den Oord et al. , 2016 ) and so on . Among them , generative adversarial networks ( GANs ) are implicit generative models ( Goodfellow et al. , 2014 ) that explicit likelihood function is not required , and are trained by playing a minimax game between the discriminator and the generator ; Variational auto-encoders ( VAEs , Kingma & Welling ( 2013 ) ; Rezende et al . ( 2014 ) ) are latent variable generative models optimized by maximizing a lower bound , called evidence lower bound , of the data log-likelihood . Flow-based models ( Dinh et al. , 2016 ; 2014 ; van den Oord et al. , 2016 ) differ from them in that they provide exact log-likelihood evaluation with change of variables theorem ( Rezende & Mohamed , 2015 ) . A flow usually starts with a simple base probability distribution , e.g . diagonal Gaussian , then follows a chain of transformations in order to approximate complex distributions . Each transformation is parameterized by specially designed neural networks so that the log-determinant of its Jacobian can be efficiently computed . Most of the previous works focus on how to design more flexible transformations to achieve tighter log-likelihoods , and generate more realistic samples . It is also believed that flows can be used to detect out-of-distribution ( OoD ) samples by assigning low likelihoods on them . However , it has been observed that flows fail to do so . For example , flows trained on FashionMNIST surprisingly assign higher likelihoods on MNIST samples ( Nalisnick et al. , 2018 ; Choi & Jang , 2018 ) . Though analyses on pixel-level statistics are performed on this phenomenon ( Nalisnick et al. , 2018 ) , and density evaluation combined with uncertainty estimation is used to detect OoD samples ( Choi & Jang , 2018 ) , the reasons behind flows ’ counter-intuitive behaviours are still not clear . Humans easily discriminate MNIST images from FashionMNIST images , since their high-level image semantics are perceptually different . Accordingly , it takes some metrics that can reflect the high-level image semantics for OoD detection . In this paper , we empirically explore the correlation between flows ’ likelihoods and image semantics , and question the rationality and applicability of using predictive likelihoods of flows for OoD detection . We first introduce a concept of semanticinvariant transformation ( SIT ) . An SIT transforms an input without changing its high-level semantics , e.g . a dog image through an SIT is still supposed to be recognized as a dog . We choose two typical flow-based models as target models : Glow ( Kingma & Dhariwal , 2018 ) , based on coupling transformations , and pixelCNN ( van den Oord et al. , 2016 ) , based on autoregressive transformations . We evaluate on image datasets MNIST and FashionMNIST under three trivial SITs : image translation , random noise perturbation , and latent factors zeroing ( specific to invertible flows using multi-scale architectures , e.g . Glow ) . We demonstrate that the predictive likelihoods of the target models show weak correlation to the image semantics in the following ways : • Small pixel translations of test images could result in obvious likelihood decreases of Glow . • Perturbing small random noises , unnoticeable to humans , to test images could lead to catas- trophic likelihood decreases of target models . This also applies even if we keep the semantic object of a test image intact , and only add noises to the background . • For an invertible flow using multi-scale architecture , e.g . Glow , the inferred latent variables of an image is a list of gaussianized and standardized factors . We find that the contributions of a flow ’ s blocks to the log-likelihood are constant and independent of inputs . Thus , simply zeroing the preceding latent factors of a sample image , and feed them to flow ’ s reverse function . We could obtain new samples with surprisingly higher likelihoods , yet with perceptually unnoticeable changes from the original image . We emphasize that all these SITs are small pixel-level modifications on test images , and undoubtedly have no influences on humans ’ recognition of the semantic objects in the images . However , they lead to obvious inconsistency of flows ’ likelihoods on test samples . Considering that the predictive likelihood of a flow is the joint probability of all the image pixels , it may not convincingly indicate the existence of a semantic object in an image . Thus it could be problematic to use flows for downstream tasks which require metrics that can reflect image semantics , e.g . OoD detection . 2 BACKGROUND . 2.1 CHANGE OF VARIABLES THEOREM . Given a random variable z with probability density function p ( z ) , after applying an invertible function f : RD → RD on z , we get a new random variable z′ = f ( z ) . Then probability density function of the changed variable z′ is given by : p ( z′ ) = p ( z ) ∣∣det ∂f−1 ∂z ∣∣ ( 1 ) We can construct arbitrarily complex probability distributions by transforming a simple base distribution p ( z0 ) with a chain of mappings fk of length K. Then we have : log p ( zK ) = log p ( z0 ) + K∑ k=1 log ∣∣det ∂f−1k ∂zk−1 ∣∣ ( 2 ) 2.2 FLOW-BASED MODELS . Flow-based models are generative models designed by applying the above theorem , thus exact loglikelihood evaluation of data is feasible . Then the practical problem of building flow-based models applied on high-dimensional data , like images , becomes how to design invertible transformations whose Jacobian determinant can be efficiently computed . Research on flows is very active and rapidly evolving . In this paper , we particularly focus on flowbased generative models on images and the behaviours of their likelihoods . They can roughly be divided into two categories according to the granularity of the transformation layers : Coupling Flow A coupling flow contains a sequence of coupling layers which model the transformation in a coarse way . Coupling layers split a D-dimensional intermediate random variable z into two parts : z1 : d , zd : D. A general form of coupling layer is affine ( Eq . 3 ) . The first part z1 : d is kept still , while the second part xd : D is scaled and shifted with transformations s , t on the first part x1 : d. y1 : d = x1 : d , yd : D = xd : D · exp ( s ( x1 : d ) ) + t ( x1 : d ) . ( 3 ) The affine coupling is proposed in Real NVP ( Dinh et al. , 2016 ) , whose Jacobian is a lower triangular matrix that can be efficiently computed . An earlier and simpler version is additive coupling proposed in NICE ( Dinh et al. , 2014 ) , which can be obtained by simply removing the scale item exp s ( x1 : d ) in affine coupling . Additive coupling layer is volume-preserving and the log-determinant of its Jacobian is always 0 . Glow improves Real NVP by replacing the fixed shuffling permutation with 1× 1 invertible convolution . Since forward and inverse operation of a coupling layer have the same computational efficiency , both likelihood evaluation and sampling ( or generation ) for coupling flows are equally efficient . Autoregressive Flow As the building blocks of autoregressive flow , autoregressive transformations model the joint probability p ( x ) as the product of one-dimensional conditionals : p ( x ) = D∏ i=1 p ( xi|x < i ) , ( 4 ) where the probability of observation xi conditions only on its previous observations x < i . The autoregressive property of an autoregressive layer is enforced by specially designed mechanism , e.g . masking . In PixelCNN ( van den Oord et al. , 2016 ) , this is implemented as masked convolutional layers , which are inherently easier to be parallelized than its counterpart PixelRNN ( Oord et al. , 2016 ) . The likelihood evaluation of PixelCNN takes only one-forward pass , but its inference , i.e . generation , takes O ( D ) , since we have to sample pixel-by-pixel . PixelCNN can be further parallelized ( Reed et al. , 2017b ) to accelerate its inference speed . 3 OTHER RELATED WORKS . Flows for generation Serving as powerful decoders , PixelCNN can also be combined with other generative models , e.g . PixelGAN ( Makhzani & Frey , 2017 ) and PixelVAE ( Gulrajani et al. , 2016 ) . Variants of PixelCNN are also used to model audio ( Van Den Oord et al. , 2016 ) , video ( Kalchbrenner et al. , 2017 ) , and text ( Kalchbrenner et al. , 2016 ) . PixelCNN , combining with attention , is also applied to few-shot autoregressive density estimation Reed et al . ( 2017a ) . Ho et al . ( 2019 ) proposes several improvements to coupling flow , reducing its gap to autoregressive flow in terms of density estimation . Autoregressive models for density estimation Autoregressive models can be specially designed for general-purpose density estimation . Masked Autoencoder for Distribution Estimation ( MADE , Germain et al . ( 2015 ) ) is a pioneering work that use masked neural networks to model the autoregressive density . MADE constitutes the building block of two popular normalizing flows : Inverse Autoregressive Flow ( IAF , Kingma et al . ( 2016 ) ) and Masked Autoregressive Flow ( MAF , Papamakarios et al . ( 2017 ) ) . IAF and MAF are similar but with different computational trade-offs . IAF , providing efficient sampling , is designed to improve the expressiveness of the approximate posterior of VAE . MAF is a more powerful density estimator which stacks multiple MADEs .

Summary: This paper raises a problem of the robustness of (log) likelihood computed by invertible flows. The authors show that the changes of likelihood of an image computed by flow-based image generative models have surprisingly weak correlations with semantic changes of image. The flow likelihoods are sensitive to very small changes of pixels that do not affect the semantics of an image. And the likelihoods are less robust against out-of-distribution inputs where we expect strong robustness compared to discriminative models.