Title: EXPLORING THE CORRELATION BETWEEN LIKELI-HOOD OF FLOW-BASED GENERATIVE MODELS AND IMAGE SEMANTICS

Abstract: Among deep generative models, flow-based models, simply referred as flows in this paper, differ from other models in that they provide tractable likelihood. Besides being an evaluation metric of synthesized data, flows are supposed to be robust against out-of-distribution (OoD) inputs since they do not discard any information of the inputs. However, it has been observed that flows trained on Fash-ionMNIST assign higher likelihoods to OoD samples from MNIST. This counterintuitive observation raises the concern about the robustness of flows' likelihood. In this paper, we explore the correlation between flows' likelihood and image semantics. We choose two typical flows as the target models: Glow, based on coupling transformations, and pixelCNN, based on autoregressive transformations. Our experiments reveal surprisingly weak correlation between flows' likelihoods and image semantics: the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invariant transformations (SITs). We explore three SITs (all small pixel-level modifications): image pixel translation, random noise perturbation, latent factors zeroing (limited to flows using multi-scale architecture, e.g. Glow). These findings, though counter-intuitive, resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels. So flows' likelihoods, modeling on pixel-level intensities, is not able to indicate the existence likelihood of the high-level image semantics. We call for attention that it may be abuse if we use the predictive likelihoods of flows for OoD samples detection.

1. INTRODUCTION:

Deep generative models have been very successful in image generation (Brock et al., 2018; Kingma & Dhariwal, 2018; Miyato et al., 2018) , natural language generation (Bowman et al., 2015; Yu et al., 2017) , audio synthesis (Van Den Oord et al., 2016) and so on. Among them, generative adversarial networks (GANs) are implicit generative models (Goodfellow et al., 2014) that explicit likelihood function is not required, and are trained by playing a minimax game between the discriminator and the generator; Variational auto-encoders (VAEs,Kingma & Welling (2013) ; Rezende et al. ( 2014)) are latent variable generative models optimized by maximizing a lower bound, called evidence lower bound, of the data log-likelihood. Flow-based models (Dinh et al., 2016; 2014; van den Oord et al., 2016) differ from them in that they provide exact log-likelihood evaluation with change of variables theorem (Rezende & Mohamed, 2015) . A flow usually starts with a simple base probability distribution, e.g. diagonal Gaussian, then follows a chain of transformations in order to approximate complex distributions. Each transformation is parameterized by specially designed neural networks so that the log-determinant of its Jacobian can be efficiently computed.

Most of the previous works focus on how to design more flexible transformations to achieve tighter log-likelihoods, and generate more realistic samples. It is also believed that flows can be used to detect out-of-distribution(OoD) samples by assigning low likelihoods on them. However, it has been observed that flows fail to do so. For example, flows trained on FashionMNIST surprisingly assign higher likelihoods on MNIST samples (Nalisnick et al., 2018; Choi & Jang, 2018) . Though analyses on pixel-level statistics are performed on this phenomenon (Nalisnick et al., 2018) , and density evaluation combined with uncertainty estimation is used to detect OoD samples (Choi & Jang, 2018) , the reasons behind flows' counter-intuitive behaviours are still not clear.

Humans easily discriminate MNIST images from FashionMNIST images, since their high-level image semantics are perceptually different. Accordingly, it takes some metrics that can reflect the high-level image semantics for OoD detection. In this paper, we empirically explore the correlation between flows' likelihoods and image semantics, and question the rationality and applicability of using predictive likelihoods of flows for OoD detection. We first introduce a concept of semanticinvariant transformation (SIT). An SIT transforms an input without changing its high-level semantics, e.g. a dog image through an SIT is still supposed to be recognized as a dog. We choose two typical flow-based models as target models: Glow (Kingma & Dhariwal, 2018) , based on coupling transformations, and pixelCNN (van den Oord et al., 2016) , based on autoregressive transformations. We evaluate on image datasets MNIST and FashionMNIST under three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (specific to invertible flows using multi-scale architectures, e.g. Glow).

We demonstrate that the predictive likelihoods of the target models show weak correlation to the image semantics in the following ways:

• Small pixel translations of test images could result in obvious likelihood decreases of Glow.

• Perturbing small random noises, unnoticeable to humans, to test images could lead to catastrophic likelihood decreases of target models. This also applies even if we keep the semantic object of a test image intact, and only add noises to the background.

• For an invertible flow using multi-scale architecture, e.g. Glow, the inferred latent variables of an image is a list of gaussianized and standardized factors. We find that the contributions of a flow's blocks to the log-likelihood are constant and independent of inputs. Thus, simply zeroing the preceding latent factors of a sample image, and feed them to flow's reverse function. We could obtain new samples with surprisingly higher likelihoods, yet with perceptually unnoticeable changes from the original image.

We emphasize that all these SITs are small pixel-level modifications on test images, and undoubtedly have no influences on humans' recognition of the semantic objects in the images. However, they lead to obvious inconsistency of flows' likelihoods on test samples. Considering that the predictive likelihood of a flow is the joint probability of all the image pixels, it may not convincingly indicate the existence of a semantic object in an image. Thus it could be problematic to use flows for downstream tasks which require metrics that can reflect image semantics, e.g. OoD detection.

2. BACKGROUND:



2.1. CHANGE OF VARIABLES THEOREM:

Given a random variable z with probability density function p(z), after applying an invertible function f : R D → R D on z, we get a new random variable z = f (z). Then probability density function of the changed variable z is given by:

p(z ) = p(z) det ∂f -1 ∂z (1)

We can construct arbitrarily complex probability distributions by transforming a simple base distribution p(z 0 ) with a chain of mappings f k of length K. Then we have :

log p(z K ) = log p(z 0 ) + K k=1 log det ∂f -1 k ∂z k-1 (2)

2.2 FLOW-BASED MODELS Flow-based models are generative models designed by applying the above theorem, thus exact loglikelihood evaluation of data is feasible. Then the practical problem of building flow-based models applied on high-dimensional data, like images, becomes how to design invertible transformations whose Jacobian determinant can be efficiently computed.

Research on flows is very active and rapidly evolving. In this paper, we particularly focus on flowbased generative models on images and the behaviours of their likelihoods. They can roughly be divided into two categories according to the granularity of the transformation layers:

Coupling Flow A coupling flow contains a sequence of coupling layers which model the transformation in a coarse way. Coupling layers split a D-dimensional intermediate random variable z into two parts: z 1:d , z d:D . A general form of coupling layer is affine(Eq. 3). The first part z 1:d is kept still, while the second part x d:D is scaled and shifted with transformations s, t on the first part x 1:d .

EQUATION

The affine coupling is proposed in Real NVP (Dinh et al., 2016) , whose Jacobian is a lower triangular matrix that can be efficiently computed. An earlier and simpler version is additive coupling proposed in NICE (Dinh et al., 2014) , which can be obtained by simply removing the scale item exp s(x 1:d ) in affine coupling. Additive coupling layer is volume-preserving and the log-determinant of its Jacobian is always 0. Glow improves Real NVP by replacing the fixed shuffling permutation with 1 × 1 invertible convolution. Since forward and inverse operation of a coupling layer have the same computational efficiency, both likelihood evaluation and sampling(or generation) for coupling flows are equally efficient.

Autoregressive Flow As the building blocks of autoregressive flow, autoregressive transformations model the joint probability p(x) as the product of one-dimensional conditionals:

EQUATION

where the probability of observation x i conditions only on its previous observations x <i . The autoregressive property of an autoregressive layer is enforced by specially designed mechanism, e.g. masking.

In PixelCNN (van den Oord et al., 2016) , this is implemented as masked convolutional layers, which are inherently easier to be parallelized than its counterpart PixelRNN (Oord et al., 2016) . The likelihood evaluation of PixelCNN takes only one-forward pass, but its inference, i.e. generation, takes O(D), since we have to sample pixel-by-pixel. PixelCNN can be further parallelized (Reed et al., 2017b) to accelerate its inference speed.

3. OTHER RELATED WORKS:

Flows for generation Serving as powerful decoders, PixelCNN can also be combined with other generative models, e.g. PixelGAN (Makhzani & Frey, 2017) and PixelVAE (Gulrajani et al., 2016) .

Variants of PixelCNN are also used to model audio (Van Den Oord et al., 2016) , video (Kalchbrenner et al., 2017) , and text (Kalchbrenner et al., 2016) . PixelCNN, combining with attention, is also applied to few-shot autoregressive density estimation Reed et al. (2017a) . Ho et al. (2019) proposes several improvements to coupling flow, reducing its gap to autoregressive flow in terms of density estimation.

Autoregressive models for density estimation Autoregressive models can be specially designed for general-purpose density estimation. 

4. METHOD:

Let g be a flow-based generative model trained on dataset X = {x i } N i=1 sampled from some unknown p(x), and p g (x) be the g's predictive probability density function of sample x .

Semantic-Invariant Transformation (SIT) Roughly speaking, SIT can be any transformation that do not change humans' recognition of image semantics. For example, suppose x is a dog image. After applying SIT T to x, T (x) is supposed to be high recognized as a dog image. As a proof of concept evaluation, we limit our evaluations to three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (specific to Glow).

Semantic Correlation:

We probe the correlation between the predictive likelihood of a flow g and image semantics by examining the influences of SITs on test samples' likelihoods. Specifically, a reasonable observation we should expect is that for SIT T : p g (T (x)) -p g (x) < δ holds for a small positive scalar δ.

We report bits-per-dim (BPD), which is given by:

BPD = NLL (h × w × c) • log 2 (5)

where NLL is the negative log-likelihood of the test sample, h, w, c are height, width, number of channels. Lower BPD implies higher likelihood. Throughout this paper, we use BPD and likelihood interchangeably. We refer to Supp. A for setup and training details of the target models.

4.1. IMAGE TRANSLATION:

Translation invariance is a fundamental property in learning image representations that are robust for downstream tasks. In this section, we evaluate the influences of image translations on flows' likelihoods. The results in Fig. 1 and examples in Fig. 2 show that even 1 or 2-pixel left translation could lead to obvious increase of Glow's predictive BPDs, while the PixelCNN's predictive BPDs are robust to pixel translation. This surprising difference can be attributed to the difference of their architectures. Glow, like other flows based on coupling transformation layers (Dinh et al., 2016) , models the joint probability of the pixels in a coarse-grained way. They rely on multi-scale architecture modeling different level of abstractions in order to achieve competitive BPDs. At higher scale levels, the intermediate tensors have smaller spatial sizes and bigger channel sizes. This is performed at the starting point of each scale level with squeeze operation, which trades spatial sizes for channel sizes by transforming a h × w × c tensor into a h/2 × w/2 × 4c tensor. Note that squeeze operation actually destroys the spatial positions of adjacent pixels, and 1-pixel translation could lead to quite different spatial partitions. While for PixelCNN, the intermediate tensors are not reshaped, and the spatial positions of pixels are kept still; Furthermore, the prediction of each pixel conditions only on a neighborhood of (previous) pixels in masked convolution, so translation invariance is preserved. Problematic Likelihood Comparisons The foundation of using likelihood-based models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples x out than indistribution samples x in , i.e. p g (x in ) p g (x out ). However, it has been observed that flows assign higher likelihoods on OoD samples than even training samples (Nalisnick et al., 2018; Choi & Jang, 2018) . Analyses of pixel-level datasets statistics in (Nalisnick et al., 2018) show that this may be due to OoD datasets just "sit inside of" in-distribution datasets with roughly the same mean, smaller variance. Surprisingly, similar counter-intuitive likelihood assignment also occurs in in-distribution samples. For example, in Fig. 1 , images with class label 1 are consistently have significantly lower BPDs, i.e. higher likelihoods than samples of other classes. In OoD detection, we assume that a sample with a higher likelihood indicates that it is more likely to be an in-distribution sample. Following the same logic, this tells that all images of class label 1 are more likely to be in-distribution samples than samples from other classes, which contradicts the fact that they are all in-distribution samples for sure. We may reasonably suspect that flows' counter-intuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics, e.g. different numbers. This kind of counter-intuitive likelihood comparisons exist not only between in-distribution and OoD samples, but also within in-distribution samples from different classes. Similarly, Theis et al. (2015) find that 1-pixel shift of the images could lead to quite different nearest neighbours from the training set, measured in Euclidean distance, which also demonstrate the gap between pixel-level metrics and humans' perception.

4.2. RANDOM NOISE PERTURBATION:

Image pixels are discrete integers; and in practice, right amount of real-valued uniform noise is added to dequantize the pixels. For images x ∈ [0, . . . , 255/256] D scaled to [0, 1], we usually do x = x + u, u ∈ [0, 1/256] D . In our experiments, we find that adding small random perturbations out of the coverage of the added noise, i.e. u ∈ [0, 1/256] D here, to test images cause flows to give catastrophically higher BPDs. Humans can robustly recognize semantic objects in images regardless of the backgrounds. So we also evaluate influences of adding random perturbations to only the backgrounds of test images. This can be simply implemented using proper mask:

x ← x + • m noise (6)

where m is the mask, and is a small scaling factor ensuring the noise is small enough.

In our evaluations, we use unit Gaussian noises, and set the scaling factor = 0.001. Examples in Fig. 3 manifest that adding small noises catastrophically lower samples' likelihoods. Compared to Glow, PixelCNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the influences of the added noise. We get similar results even if we keep the semantic objects of test images intact, and add noises only to the backgrounds. Note that the Gaussian noises • N (0, 1) we added are out of the coverage (with < 0 elements) of the uniform noises [0, 1/256] added during training, so theoretically this is expected since models are not optimized in that areas. However, it does reveal that flows are not aware of the image semantics, and treat the pixels of objects and the pixels of backgrounds with no discrimination. Other tested noises include •(1/256+ N (0, 1)) and • [-1/256, 0], and similar results were obtained.

4.3. FREE LIKELIHOOD LOOPHOLE OF GLOW:

Algorithm 1 Generate x * by zeroing the latent factors

1: Input: image x, label y (optional), Glow model g, k < L. 2: z ← g.forward(x, y)

Infer the latent factors. 3: z * ← zero(z, k)

Zero the preceding k factors. 4: x * ← g.reverse(z * , y)

Reverse the zeroed latent factors. 5: return x * Let us first decompose the Glow architecture into blocks and review their contributions to the final log-likelihood. A Glow consists of a sequence of modules at different scale levels. At each scale level, it starts with a squeeze operation, which reshapes the intermediate tensor without contribution to the log-likelihood. Following each squeeze operation is a stack of step flow blocks. A step flow block consists of three layers: actnorm, invertible 1 × 1 convolutional layer, and coupling layer. The log-determinants of actnorm and 1×1 convolutional layer are input-independent, and depend only on their inner weights (see Table 1 in (Kingma & Dhariwal, 2018) ). Additive coupling layer is volumepreserving whose log-determinant is 0, thus is also input-independent. For affine coupling layer, its log-determinant depends on the affined half, but is quantitatively small. Compared to Glow with additive coupling layers, affine layers bring only a small improvement of < 0.05 BPD (Kingma & Dhariwal, 2018) . Then, the intermediary tensor is split into two halves along the channel dimension. One halve is gaussianized with a convolutional block, then the other halve is factored out after being standardized. This procedure significantly reduces the amount of computation and memory. We refer to Dinh et al. (2016) for more details due to limited space. So for a Glow using additive couplings, the cumulative log-determinant of the flow blocks within a particular scale-level is constant regardless of different samples. Only the log-determinants of gaussianized factorings (between the transitions of different scale-levels) depend on the individual inputs. Denote z = {z 1 , . . . , z L } as the latent variables of input image x. Each z i is the standardized vector of i-th scale level. This simply means that a sample x whose latent variables z are close to center 0 will have a higher log-likelihood. Empirically, it also applies to Glow using affine coupling layers, since the influence of varying the latent variables on the log-determinants of affine coupling layers is quantitatively small compared to its gain.

We could make use of this property to generate samples with higher likelihoods via the invertibility of Glow for free (see Algorithm 1, label y is optional). We find that the semantic object of a test image depends heavily on the last factored latent z L , rather than the preceding factors. Examples in Fig. 5 show that zeroing the preceding 1 and 2 latent factors could give us samples with obviously lower BPDs but without obvious changes (slightly faded pixel intensities) of the semantic objects. Results in Fig. 4 show that zeroing the first latent factor give the maximum increment of likelihoods.

4.4. IMPLICATIONS ON DISCRIMINATIVE CLASSIFIERS:

We also evaluate the influences of these SITs on the performance of discriminative classifiers. In contrast to the obvious change of flows' likelihoods, these small perturbations could decrease the testing accuracies of classifiers to some insignificant, or negligible (on MNIST) extent (see Tab 1). The discriminative classifier used here is a shallow residual network of 8 layers on 32 × 32 images in the structure specified in (He et al., 2016) Difference to Adversarial Examples Both the SITs we use in this paper and adversarial perturbations are small perturbations, but they are inherently different. Adversarial perturbations are intentionally crafted to cause misbehaviors, which are usually specific to individual images, and easily fool a classifier with almost 0% accuracy. While three SITs above are universal transformations over all images that take no additional computations and basically come for free.

5. DISCUSSIONS AND CONCLUSIONS:

What is the problem of likelihood-based generative models? Discriminative classifiers, trained to extract class-relevant features, are known to be vulnerable to adversarial examples, and give over-confident predictions even for OoD samples. Generative models are supposed to be more robust since they model every pixel information of an image. However, likelihood modeling in high-dimensional space can be hard and lead to counter-intuitive observations. It was observed that likelihood-based generative models can assign even higher likelihoods on OoD samples (Nalisnick et al., 2018; Choi & Jang, 2018) . Nalisnick et al. (2018) observe this phenomenon on both flows and VAEs. They decompose the change-of-variable theorem and investigate the influences of different transformation layers, find that the phenomenon still exists regardless of whether the transformation is volume-preserving or not. Their second-order analysis on pixel statistics suggests that OoD datasets, e.g. MNIST, just sit inside of in-distribution datasets, e.g. FashinMNIST, with roughly the same mean, smaller variance. They suspect that flows may simply fit the pixel intensities without really capture the high-level semantics. Ren (2019) find that the likelihood of an image is mostly dominated by the irrelevant background pixels, and propose a remedy to correct the original likelihood with a likelihood ratio. Though significantly improves the accuracy of OoD detection, but still fail to answer the question: whether the likelihood ratio shows high correlation to high-level semantics.

This paper differs from previous works and step further to explore the correlations between the likelihood of flow-based generative models and image semantics. Theoretical analyses in (Theis et al., 2015; van den Oord & Dambre, 2015) point out an important argument that generative models' ability to produce plausible samples is neither sufficient nor necessary for high likelihood. Results in this paper provide more experimental evidences for this simple argument that even for powerful exact likelihood-based generative models-flows, the likelihoods of samples can be largely weakly correlated to the high-level image semantics. Thus, special attention should be paid to this argument before we apply likelihood-based generative models to downstream tasks. For example, considering the weak correlation between flows' likelihoods and image semantics, it may be inappropriate to use them for OoD samples detection. On the other hand, these counter-intuitive behaviours of flows raise our awareness of the gap between the predictive likelihoods of flows and the expectation that these likelihoods can closely relate to the semantics for OoD detection.

What is exactly the likelihood of a image? We should keep in mind that the predictive likelihood of a flow is the joint probability of all the image pixels. There is no doubt that flows, trained by maximizing its likelihood, could generate impressive synthesized data. There seem to be no problem that in terms of image generation, we expect that every single generated pixel in a image is the most likely one (hinging on its contextual pixels). However, the likelihood is explicitly modeled on pixels, so can be easily influenced by pixel-level modifications. Images' likelihoods significantly decrease even small noises are added to the pixels of backgrounds. For downstream tasks that need some "likelihood" to indicate the object in an image is a cat, rather than a car, the pixels of backgrounds are almost irrelevant. This drive us to think that we may need to model likelihood in some kind of semantic space or with some "perceptual" metrics, rather than on raw pixels. One promising direction is to define likelihood of an images on its high-level representation, and successful examples are (Lee, 2018; Nilesh A. Ahuja, 2019) . 

:



Figures and Tables:

Figure 1: BPDs of (affine) Glow and PixelCNN on MNIST test set with 1-pixel and 2-pixel left translation. Shown are class labels from 0 to 4. See Supp. B.1 for additional results.

Figure 2: Examples from MNIST, FashionMNIST test sets with 1-pixel and 2-pixel left translations. Below are predictive BPDs of Glows (Affine and Additive) and PixelCNN.

Figure 3: Examples from MNIST, FashionMNIST test sets perturbed with gaissian noises. The middle column shows the original images. The second and fourth columns are the gaussian noises and masked gaussian noises. The first and fifth columns are the perturbed images with noises w/o (+/-) masks. Below the samples are the BPDs reported by Affine Glow and PixelCNN. Test images are scaled to [0, 1]. We could simply generate mask by setting a threshold for the scaled pixels. Here we use m = x < 0.3. See Supp. D for more examples.

Figure 4: BPDs of affine and additive Glow on MNIST, FashionMNIST test sets by zeroing the preceding 1 (zero z ≤1 ) and 2 (zero z ≤2 ) latent factors. Shown are classes 0 to 4. See Supp. C for additional results.

Figure 7: BPDs of (additive) Glow on MNIST and FashionMNIST test sets with different 1-pixel and 2-pixel left translation (1p-left, 2p-left).

Figure 11: Examples from FashionMNIST test sets perturbed with gaissian noises. The middle column shows the original images. The second and fourth columns are the gaussian noises and masked gaussian noises. The first and fifth columns are the perturbed images with noises w/o (+/-) masks. Below the samples are the BPDs reported by Affine Glow and PixelCNN.

. Examples of Affine Glow with different zeroed latent variables on MNIST (left) and FashionMNIST (right) test sets. See Supp. C.1 for results of Additive Glow. Test accuracies of discriminative classifiers on MNIST, FashionMNIST test sets with different perturbations.

Table content: <table><tr><td/><td>original</td><td>zero z ≤1</td><td colspan="2">zero z ≤2</td><td>original</td><td>zero z ≤1</td><td>zero z ≤2</td></tr><tr><td>Glow</td><td>0.852</td><td>0.645</td><td/><td>0.588</td><td>1.715</td><td>1.368</td><td>1.356</td></tr><tr><td>Glow</td><td>1.306</td><td>1.062</td><td/><td>1.009</td><td>2.967</td><td>2.526</td><td>2.508</td></tr><tr><td colspan="4">Figure 5: Perturbations</td><td colspan="2">MNIST FashionMNIST</td><td/></tr><tr><td/><td/><td>Clean</td><td/><td>99.31%</td><td>93.03%</td><td/></tr><tr><td/><td/><td>1-pixel left</td><td/><td>99.12%</td><td>89.71%</td><td/></tr><tr><td/><td/><td colspan="3">Gaussian noises 99.31%</td><td>93.04%</td><td/></tr><tr><td/><td/><td>Zero &lt;= 1</td><td/><td>99.18%</td><td>91.44%</td><td/></tr><tr><td/><td/><td>Zero &lt;= 2</td><td/><td>99.07%</td><td>86.22%</td><td/></tr></table>

Figure 10: Examples from MNIST test sets perturbed with gaissian noises. The middle column shows the original images. The second and fourth columns are the gaussian noises and masked gaussian noises. The first and fifth columns are the perturbed images with noises w/o (+/-) masks. Below the samples are the BPDs reported by Affine Glow and PixelCNN.

Table content: <table><tr><td colspan="5">C.1 EXAMPLES OF ADDITIVE GLOW</td><td/><td/><td/><td/></tr><tr><td colspan="4">original perturbed/-zero z ≤1</td><td>zero z ≤2 noise</td><td colspan="4">original original masked noise perturbed/+ zero z ≤1 zero z ≤2</td></tr><tr><td>Glow Glow</td><td>0.981</td><td>2.389</td><td>0.608</td><td>0.493</td><td>1.285</td><td>1.761</td><td>1.453</td><td>1.432 2.199</td></tr><tr><td>PixelCNN</td><td/><td>3.413</td><td/><td/><td>1.131</td><td/><td/><td>3.113</td></tr><tr><td>Glow</td><td>1.483</td><td/><td>1.111</td><td>1.013</td><td/><td>3.217</td><td>2.852</td><td>2.772</td></tr><tr><td>Glow</td><td/><td>2.113</td><td/><td/><td>1.102</td><td/><td/><td>2.161</td></tr><tr><td>PixelCNN</td><td/><td>2.952</td><td/><td/><td>1.063</td><td/><td/><td>3.180</td></tr><tr><td>Glow</td><td>0.423</td><td/><td>0.071</td><td>0.034</td><td/><td>1.314</td><td>1.068</td><td>1.033</td></tr><tr><td/><td colspan="3">perturbed/-</td><td>noise</td><td>original</td><td colspan="3">masked noise perturbed/+</td></tr><tr><td>Glow</td><td/><td>2.966</td><td/><td/><td>2.505</td><td/><td/><td>2.966</td></tr><tr><td>PixelCNN</td><td/><td>3.359</td><td/><td/><td>2.304</td><td/><td/><td>3.153</td></tr><tr><td>Glow</td><td/><td>2.969</td><td/><td/><td>2.274</td><td/><td/><td>3.042</td></tr><tr><td>PixelCNN</td><td/><td>3.202</td><td/><td/><td>2.215</td><td/><td/><td>3.204</td></tr></table>

Peer review (iclr2020):
Summary: The paper studies likelihood-based models of images, such as Glow and PixelCNN. The paper shows empirically that image transformations that preserve semantics (e.g. translations by a few pixels) produce images that have lower probability (density) under such models. Decision: The paper is studying an important topic, which is how to use likelihood-based models correctly, and it warns against the misuse of such models. I agree with the paper*s conclusion that we should be careful when using likelihood-based models. Nonetheless, in my opinion the paper is a clear reject. The paper is full of flaws, incorrect statements, poorly constructed arguments, speculative explanations, and superficial descriptions of previous work. Broadly, the main issue with the paper is the following: (a) It begins with flawed assumptions about how a likelihood-based model is expected to behave. (b) It tests two likelihood-based models experimentally and finds that they don*t behave according to the assumptions. (c) It concludes that we need to be cautious when using likelihood-based models. As I said, I agree that we should be careful when using likelihood-based models, but I worry that the way the paper reaches this conclusion can mislead and misinform readers. In what follows, I elaborate on specific issues with the paper in more detail. Issue #1: The main flaw of the paper is the assumption that semantic-preserving transformations shouldn*t reduce the likelihood of the model (beginning of section 4). This is incorrect. To see why, consider a semantic-preserving transformation x* = T(x). As defined in the paper, a semantic-preserving transformation is one that doesn*t change the label y of an image x. This can be formalized as: p(y | x*) = p(y | x) By Bayes* rule, from the above it follows that: p(x* | y) p(y) / p(x*) = p(x | y) p(y) / p(x) => p(x* | y) / p(x*) = p(x | y) / p(x) Clearly p(x*) can be different from p(x), as long as p(x* | y) is different from p(x | y) by the same factor. Hence, it doesn*t follow that if p(y | x*) = p(y | x) then p(x*) = p(x), which is what the paper incorrectly assumes. To be clear, in the above expressions, p() refers to the true data distribution and not to a model that approximates it. For example, consider images of digits, where the digit is generally in the centre of the image. Moving a digit to the corner will result in a less likely image, because it*s unlikely that digits appear in corners. However, it won*t change the classification of that digit, since all digits are less likely to appear in corners in exactly the same way. In fact, this is exactly why we see in section 4.1 that the likelihood of the model decreases as the image is translated to the left; the model is behaving exactly as it is supposed to. Similarly, in section 4.2 where noise is added to the image, the model is again behaving exactly as it is supposed to. Adding Gaussian noise to the image results in a distribution p(x*) that is equal to convolving the original distribution p(x) with the noise distribution p(noise) which is an isotropic Gaussian. As a result, p(x*) will be a more diffuse version of p(x), hence samples x* will have on average low probability (density) under p(x), exactly as expected, and exactly as the experiment observes. Issue #2: The paper incorrectly assumes that out-of-distribution examples should have low probability (density). This is incorrect, and a common misconception that results from confusing high probability with typicality. In fact, out-of-distribution examples can have high probability (density). Here are two examples that illustrate that: Suppose you flip a bent coin a million times, with 10% probability of the coin coming up heads. The in-distribution samples (the typical set) are those sequences of coin tosses that have roughly 100 thousand heads. However, the most likely outcome is the all-tails sequence. This outcome is clearly atypical, and many people would agree that it*s out-of-distribution, but it has the highest probability. Consider one million independent Gaussian variables, each with mean 0 and variance 1. Due to the law of large numbers, a typical draw of these variables will have average squared value very close to 1, hence the outcome of all variables being zero is very atypical and many people would agree it*s out-of-distribution. However, the all-zero outcome is in fact the one with the highest probability density. Given the above, the following two statements copied from the paper are flawed, and potentially misleading: *The foundation of using likelihood-based models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples than in-distribution samples* *In OoD detection, we assume that a sample with a higher likelihood indicates that it is more likely to be an in-distribution sample* Fundamentally, I think the issue is that the paper incorrectly assumes that all images with the same semantics (e.g. all images of the digit 3) must be in-distribution. However this is not necessarily true. For example, the true data-generating process of MNIST images (i.e. asking people to write down a digit, scanning it, denoising it, cropping it and centring it) is unlikely to produce images where the digit is not in the centre or the background is noisy. Hence, the images considered in sections 4.1 and 4.2 are indeed out-of-distribution with respect to the true data distribution of MNIST, and are not adversarial examples of the models as the paper suggests. Further issues: The paper is ostensibly about flow models, but in fact very little is specific to flow models, and most of the discussion, where correct, applies to likelihood-based models in general. In fact, PixelCNN is not a flow model, even though the paper misleadingly describes it as such. PixelCNN can be used to model discrete random variables, whereas flow models are used for continuous random variables (flows for discrete random variables exist, but they are different from PixelCNN). That said, if PixelCNN is used to model continuous random variables then by reparameterization it can be viewed as a flow model with one layer, but that would be an unusual way to present it. Same for WaveNet. *It is also believed that flows can be used to detect out-of-distribution(OoD) samples by assigning low likelihoods on them.* Believed by whom? A citation is needed here. *Flows can roughly be divided into two categories [coupling flows and autoregressive flows]* There are several flows that fall in neither of these categories, such as linear flows, residual flows, planar flows, radial flows, Sylvester flows, neural ODEs, FFJORD, and many others. *The autoregressive property of an autoregressive layer is enforced by masking.* There are other ways of enforcing the autoregressive property (e.g. RNNs); masking is just one of them. Eq. (5) is not correct in general. The bits per dimension should be approximated by: BPD = (NLL - log|B|) / ((h x w x c) * log2) where |B| is the quantization volume, provided |B| is small. That is, if each pixel with range [0, 1] is quantized into 10 bins, then |B| = 0.1 ^ (h x w x c). *This surprising difference can be attributed to the difference of their architectures* (beginning of page 5) The explanation that follows is speculative, but is not presented as such, which can be misleading. *We may reasonably suspect that flows’ counter-intuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics* This is also speculation. *PixelCNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the influences of the added noise* Also speculation. *We find that the semantic object of a test image depends heavily on the last factored latent zL, rather than the preceding factors* As far as I can see, there isn*t evidence in support of that statement in the paper. *Considering the weak correlation between flows’ likelihoods and image semantics, it is inappropriate to use them for OoD samples detection* Given the flawed assumption about the role of image semantics, I don*t think there is evidence for that. *In terms of image generation, we expect that every single generated pixel in a image is the most likely one* This is inaccurate; when generating images from a model, we don*t get the most likely pixels, but samples from the joint distribution over pixels.