Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models

1 INTRODUCTION . Assessing whether input data is novel or significantly different than the one used in training is critical for real-world machine learning applications . Such data are known as out-of-distribution ( OOD ) inputs , and detecting them should facilitate safe and reliable model operation . This is particularly necessary for deep neural network classifiers , which can be easily fooled by OOD data ( Nguyen et al. , 2015 ) . Several approaches have been proposed for OOD detection on top of or within a neural network classifier ( Hendrycks & Gimpel , 2017 ; Lakshminarayanan et al. , 2017 ; Liang et al. , 2018 ; Lee et al. , 2018 ) . Nonetheless , OOD detection is not limited to classification tasks nor to labeled data sets . Two examples of that are novelty detection from an unlabeled data set and next-frame prediction from video sequences . A rather obvious strategy to perform OOD detection in the absence of labels ( and even in the presence of them ) is to learn a density modelM that approximates the true distribution p∗ ( X ) of training inputs x ∈ X ( Bishop , 1994 ) . Then , if such approximation is good enough , that is , p ( x|M ) ≈ p∗ ( x ) , OOD inputs should yield a low likelihood under modelM . With complex data like audio or images , this strategy was long thought to be unattainable due to the difficulty of learning a sufficiently good model . However , with current approaches , we start having generative models that are able to learn good approximations of the density conveyed by those complex data . Autoregressive and invertible models such as PixelCNN++ ( Salimans et al. , 2017 ) and Glow ( Kingma & Dhariwal , 2018 ) perform well in this regard and , in addition , can approximate p ( x|M ) with arbitrary accuracy . Recent works , however , have shown that likelihoods derived from generative models fail to distinguish between training data and some OOD input types ( Choi et al. , 2018 ; Nalisnick et al. , 2019a ; Hendrycks et al. , 2019 ) . This occurs for different likelihood-based generative models , and even when inputs are unrelated to training data or have totally different semantics . For instance , when trained on CIFAR10 , generative models report higher likelihoods for SVHN than for CIFAR10 itself ( Fig . 1 ; data descriptions are available in Appendix A ) . Intriguingly , this behavior is not consistent across data sets , as other ones correctly tend to produce likelihoods lower than the ones of the training data ( see the example of TrafficSign in Fig . 1 ) . A number of explanations have been suggested for the root cause of this behavior ( Choi et al. , 2018 ; Nalisnick et al. , 2019a ; Ren et al. , 2019 ) but , to date , a full understanding of the phenomenon remains elusive . In this paper , we shed light to the above phenomenon , showing that likelihoods computed from generative models exhibit a strong bias towards the complexity of the corresponding inputs . We find that qualitatively complex images tend to produce the lowest likelihoods , and that simple images always yield the highest ones . In fact , we show a clear negative correlation between quantitative estimates of complexity and the likelihood of generative models . In the second part of the paper , we propose to leverage such estimates of complexity to detect OOD inputs . To do so , we introduce a widely-applicable OOD score for individual inputs that corresponds , conceptually , to a likelihoodratio test statistic . We show that such score turns likelihood-based generative models into practical and effective OOD detectors , with performances comparable to , or even better than the state-of-theart . We base our experiments on an extensive collection of alternatives , including a pool of 12 data sets , two conceptually-different generative models , increasing model sizes , and three variants of complexity estimates . 2 COMPLEXITY BIAS IN LIKELIHOOD-BASED GENERATIVE MODELS . From now on , we shall consider the log-likelihood of an input x given a model M : ` M ( x ) = log2 p ( x|M ) . Following common practice in evaluating generative models , negative log-likelihoods − ` M will be expressed in bits per dimension ( Theis et al. , 2016 ) , where dimension corresponds to the total size of x ( we resize all images to 3×32×32 pixels ) . Note that the qualitative behavior of log-likelihoods is the same as likelihoods : ideally , OOD inputs should have a low ` M , while in-distribution data should have a larger ` M. Most literature compares likelihoods of a given model for a few data sets . However , if we consider several different data sets at once and study their likelihoods , we can get some insight . In Fig . 2 , we show the log-likelihood distributions for the considered data sets ( Appendix A ) , computed with a Glow model trained on CIFAR10 . We observe that the data set with a higher log-likelihood is Constant , a data set of constant-color images , followed by Omniglot , MNIST , and FashionMNIST ; all of those featuring gray-scale images with a large presence of empty black background . On the other side of the spectrum , we observe that the data set with a lower log-likelihood is Noise , a data set of uniform random images , followed by TrafficSign and TinyImageNet ; both featuring colorful images with non-trivial background . Such ordering is perhaps more clear by looking at the average log-likelihood of each data set ( Appendix D ) . If we think about the visual complexity of the images in those data sets , it would seem that log-likelihoods tend to grow when images become simpler and with less information or content . To further confirm the previous observation , we design a controlled experiment where we can set different decreasing levels of image complexity . We train a generative model with some data set , as before , but now compute likelihoods of progressively simpler inputs . Such inputs are obtained by average-pooling the uniform random Noise images by factors of 1 , 2 , 4 , 8 , 16 , and 32 , and re-scaling back the images to the original size by nearest-neighbor up-sampling . Intuitively , a noise image with a pooling size of 1 ( no pooling ) has the highest complexity , while a noise image with a pooling of 32 ( constant-color image ) has the lowest complexity . Pooling factors from 2 to 16 then account for intermediate , decreasing levels of complexity . The result of the experiment is a progressive growing of the log-likelihood ` M ( Fig . 3 ) . Given that the only difference between data is the pooling factor , we can infer that image complexity plays a major role in generative models ’ likelihoods . Until now , we have consciously avoided a quantitative definition of complexity . However , to further study the observed phenomenon , and despite the difficulty in quantifying the multiple aspects that affect the complexity of an input ( cf . Lloyd , 2001 ) , we have to adopt one . A sensible choice would be to exploit the notion of Kolmogorov complexity ( Kolmogorov , 1963 ) which , unfortunately , is noncomputable . In such cases , we have to deal with it by calculating an upper bound using a lossless compression algorithm ( Cover & Thomas , 2006 ) . Given a set of inputs x coded with the same bit depth , the normalized size of their compressed versions , L ( x ) ( in bits per dimension ) , can be considered a reasonable estimate of their complexity . That is , given the same coding depth , a highly complex input will require more bits per dimension , while a less complex one will be compressed with fewer bits per dimension . For images , we can use PNG , JPEG2000 , or FLIF compressors ( Appendix C ) . For other data types such as audio or text , other lossless compressors should be available to produce a similar estimate . If we study the relation between generative models ’ likelihoods and our complexity estimates , we observe that there is a clear negative correlation ( Fig . 4 ) . Considering all data sets , we find Pearson ’ s correlation coefficients below −0.75 for models trained on FashionMNIST , and below −0.9 for models trained on CIFAR10 , independently of the compressor used ( Appendix D ) . Such significant correlations , all of them with infinitesimal p-values , indicate that likelihood-based measures are highly influenced by the complexity of the input image , and that this concept accounts for most of their variance . In fact , such strong correlations suggest that one may replace the computed likelihood values for the negative of the complexity estimate and obtain almost the same result ( Appendix D ) . This implies that , in terms of detecting OOD inputs , a complexity estimate would perform as well ( or bad ) as the likelihoods computed from our generative models . 3 TESTING OUT-OF-DISTRIBUTION INPUTS . 3.1 DEFINITION . As complexity seems to account for most of the variability in generative models ’ likelihoods , we propose to compensate for it when testing for possible OOD inputs . Given that both negative loglikelihoods − ` M ( x ) and the complexity estimate L ( x ) are expressed in bits per dimension ( Sec . 2 ) , we can express our OOD score as a subtraction between the two : S ( x ) = − ` M ( x ) − L ( x ) . ( 1 ) Notice that , since we use negative log-likelihoods , the higher the S , the more OOD the input x will be ( see below ) . 3.2 INTERPRETATION : OCCAM ’ S RAZOR AND THE OUT-OF-DISTRIBUTION PROBLEM . Interestingly , S can be interpreted as a likelihood-ratio test statistic . For that , we take the point of view of Bayesian model comparison or minimum description length principle ( MacKay , 2003 ) . We can think of a compressor M0 as a universal model , adjusted for all possible inputs and general enough so that it is not biased towards a particular type of data semantics . Considering the probabilistic model associated with the size of the output produced by the lossless compressor , we have p ( x|M0 ) = 2−L ( x ) and , correspondingly , L ( x ) = − log2 p ( x|M0 ) . ( 2 ) In Bayesian model comparison , we are interested in comparing the posterior probabilities of different models in light of data X . In our setting , the trained generative modelM is a ‘ simpler ’ version of the universal modelM0 , targeted to a specific semantics or data type . With it , one aims to approximate the marginal likelihood ( or model evidence ) for x ∈ X , which integrates out all model parameters : p ( x|M ) = ∫ p ( x|θ , M ) p ( θ|M ) dθ . This integral is intractable , but current generative models can approximate p ( x|M ) with arbitrary accuracy ( Kingma & Dhariwal , 2018 ) . Choosing between one or another model is then reduced to a simple likelihood ratio : log2 p ( M0|x ) p ( M|x ) = log2 p ( x|M0 ) p ( M0 ) p ( x|M ) p ( M ) . ( 3 ) For uniform priors p ( M0 ) = p ( M ) = 1/2 , this ratio is reduced to S ( x ) = − log2 p ( x|M ) + log2 p ( x|M0 ) which , using Eq . 2 for the last term , becomes Eq . 1 . The ratio S accommodates the Occam ’ s razor principle . Consider simple inputs that can be easily compressed by M0 using a few bits , and that are not present in the training of M. These cases have a high probability under M0 , effectively correcting the abnormal high likelihood given by the learned modelM . The same effect will occur with complex inputs that are not present in the training data . In these cases , both likelihoods will be low , but the universal lossless compressorM0 will predict those better than the learned modelM . The two situations will lead to large values of S. In contrast , inputs that belong to the data used to train the generative modelM will always be better predicted byM than byM0 , resulting in lower values of S .

Summary: The paper discusses the inductive biases in generative models that tend to assign higher likelihoods to "less complex" images. In particular, a likelihood based generative model (like Glow or PixelCNN++) trained on a particular dataset has significantly higher likelihood for data that have lower compression ratios (e.g  with PNG). The authors propose a simple approach based on the likelihood ratio between a trained model and a "prior" model (based on existing compression methods) and demonstrate that this improves unsupervised OOD detection on certain dataset pairs. The idea is quite simple (and surprising it seems to work!), but it seems that better understanding of the proposed method could be achieved.