Exploring the Correlation between Likelihood of Flow-based Generative Models and Image Semantics
Authors: Xin WANG
SiuMing Yiu
Venue:  (2020)
Keywords: flow-based generative models
out-of-distribution samples detection
likelihood robustness

Abstract:  Among deep generative models, flow-based models, simply referred as \emph{flow}s in this paper, differ from other models in that they provide tractable likelihood. Besides being an evaluation metric of synthesized data, flows are supposed to be robust against out-of-distribution~(OoD) inputs since they do not discard any information of the inputs. However, it has been observed that flows trained on FashionMNIST assign higher likelihoods to OoD samples from MNIST. This counter-intuitive observation raises the concern about the robustness of flows' likelihood. In this paper, we explore the correlation between flows' likelihood and image semantics. We choose two typical flows as the target models: Glow, based on coupling transformations, and pixelCNN, based on autoregressive transformations. Our experiments reveal surprisingly weak correlation between flows' likelihoods and image semantics: the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invariant transformations~(SITs). We explore three SITs~(all small pixel-level modifications): image pixel translation, random noise perturbation, latent factors zeroing~(limited to flows using multi-scale architecture, e.g. Glow). These findings, though counter-intuitive, resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels. So flows' likelihoods, modeling on pixel-level intensities, is not able to indicate the existence likelihood of the high-level image semantics. We call for attention that it may be \emph{abuse} if we use the predictive likelihoods of flows for OoD samples detection.

Under review as a conference paper at ICLR 2020
EXPLORING THE CORRELATION BETWEEN LIKELI- HOOD OF FLOW-BASED GENERATIVE MODELS AND IMAGE SEMANTICS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Among deep generative models. flow-based models, simply referred as flows in this paper, differ from other models in that they provide tractable likelihood. Be- sides being evaluation metric of synthesized data, flows are supposed to be robust against iitoo--distribution (OoD) inputs since they do not discard any in- formation of the inputs. However, it has been observed that flows trained Fash- ionMNIST assign higher likelihoods to OoD samples from MNIST. This counter- intuitive observation raises the concern about the robustness of flows' likelihood. In this paper, we explore the correlation between flows' likelihood and image se- mantics. We choose two typical flows the target models: Glow, based cou- pling transformations, and piXeICNN, based on autoregressive transformations. Our experiments reveal surprisingly weak correlation between flows' likelihoods and image semantics: the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invarian transformations (SITs). We explore three SIT's (all small pixel-level modifications): image pixel translation, random noise perturbation, la- tent factors zeroing (limited to flows using multi-scale architecture, e.g. Glow). These findings, though sounter-intuitive. resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels. So flows' like- lihoods, modeling on pixel-level intensities, not able to indicate the existence likelihood of the high-level image semantics. We call for attention that it may be abuse we use the predictive likelihoods flows for OoD samples detection.
INTRODUCTION
Deep generative models have been very successful image generation (Brock et al. 2018 8 Kingma & Dhariwal 2018 Miyato al.) 2018), natural language generation (Bowman al. 2015 Yu et al. 2017). audio synthesis Van Den Oord et al., 2016) and so on. Among them, generative adversarial networks (GANs) are implicit generative
Models(GGoddfllloo et al.] 2014) that explicit likelihood function is not required, and are trained by playing a minimax game between the discriminator and the generator; Variational auto-encoders VAEs/Kingmi are latent variable generative models optimized by maximizing a lower bound, called evidence lower bound, of the data log-likelihood. Flow-based models (Dinh et al., 2016 2014; van den Oord al., 2016) differ from them in that they provide exact log- likelioodood evaluation with change of variables theorem (Rezende & Mohamed 2013). A flow usually starts with a simple base probability dis- tribution, e.g. diagonal Gaussian, then follows a chain of transformations in order to approximate complex distributions. Fach transformation is parameterized by specially designed neural networks so that the fog-determinant of its Jacobian can be efficiently computed.
Welling (2013): Rezende al. (2014))
Most of the previous works focus on how to design more flexible transformations to achieve tighter log-likelihoods, and generate more realistic samples. It is also believed that flows can be used to detect cuttof--ditributtonnoooll samples by assigning low likelihoods on them. However, it has been observed that flows fail to do For example, flows trained FashionMNIIST surprisingly assign higher likelihoods on MNIST samples (Nalismick et al. 2018:; Choi & Jang, 2018). Though analyses on pixel-level statistics are performed on this phenomenon (Nalisnick et al. 2018). and

---

Under review as a conference paper at ICLR 2020
density evaluation combined with uncertainty estimation is used detect OoD samples (Choi & Jang 2018). the reasons behind flows' counter-intuitive behaviours are still not clear.
Humans easily discriminate MNIST images from FashionMNIST images, since their high-level im- age semantics are perceptually different. Accordingly, it takes some metrics that can reflect the high-level image semantics for OoD detection. In this paper, empirically explore the correlation between flows' likelihoods and image semantics, and question the rationality and applicability of using predictive likelihoods of flows for OoD detection. We first introduce concept of semantic- invariant transformation (SIT). An SIT transforms an input without changing its high-level seman- ties, e.g. dog image through an SIT is still supposed to be recogmized typical flow-based models as target models: Glow (Kingma & Dhariwal 2018), based on coupling transformations, and pixelCNN (van den Oord et al.J (016), based on autoregressive transforma- tions. We evaluate on image datasets MNIST and FashionMNIST under three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (specific invertible flows using multi-scale architectures, e.g. Glow).
dog. We choose two
We demonstrate that the predictive likelihoods of the target models show weak correlation to the image semantics in the following ways:
Small pixel translations of test images could result in obvious likelihood decreases of Glow.
Perturbing small random noises, unnoticeable to humans, to test images could lead to catas- trophic likelihood decreases of target models. This also applies even we keep the seman- tic object of a test image intact, and only add noises to the background.
For invertible flow using multi-scale architecture, e.g. Glow, the inferred latent variables an image is list of gaussianized and standardized factors. We find that the contributions of flow's 's blocks to the log-likelihood are constant and independent of inputs. Thus, simply zeroing the preceding latent factors a sample image, and feed them to flow's reverse function. We could obtain new samples with surprisingly higher likelihoods, yet with perceptually unnoticeable changes from the original image.
We emphasize that these SIT' are small pixel-level modifications on test images, and undoubt- edly have no influences on humans' recognition of the semantic objects in the images. However, they lead to obvious inconsistency of flows' likelihoods on test samples. Considering that the pre- dictive likelihood of flow is the joint probability of all the image pixels, it may not convincingly indicate the existence of a semantic object in an image. Thus could be problematic to use flows for downstream tasks which require metrics that reflect image semanties, e.g. OoD detection.
BACKGROUND
2.1 CHANGE OF VARIABLES THEOREM
Given random variable with probability density function p(z), after applying invertible fune- tion f : RD + RD on 2, we get new random variable z' f(z). Then probability density function of the changed variable z' is given by:
05-1
p(z') = p(=)| det
8z
We can construct arbitrarily complex probability distributions by transforming a simple base distri- bution p(zp) with chain of mappings f& of length K. Then we have
K
05
logp(zx) = log 1(z)()))
+
log det
(2)
0z
2.2 FLOW-BASED MODELS
Flow-based models are generative models designed by applying the above theorem, thus exact log- likelihood evaluation of data is feasible. Then the practical problem of building flow-hased models
2

---

Under review as a conference paper at ICLR 2020
applied ligh-dimensional data, like images, becomes how to design invertible transformations whose Jacobian determinant can be efficiently computed.
Research flows is very active and rapidly evolving. In this paper, particularly focus flow- based generative models on images and the behaviours of their likelihoods. They can roughly be divided into two categories according to the granularity of the transformation layers:
Coupling Flow A coupling flow contains sequence of coupling layers which model the transfor-
mation in coarse way. Coupling layers split a DasitaD-aaae D-dimensional intermediate random variable into two parts: 21:d, Zd:D- A general form of coupling layer is affine(Eq. 3 The first part 21:d is kept still, while the second part a is scaled and shifted with transformations s, t on the first part i:d-
Y1:d F1:d,
yd: FddDD exp(s((11a)) + ((1id))
The affine coupling is proposed in Real NVP (Dinh et al. 2016), whose Jacobian is lower triangular matrix that can efficiently computed. An earlier and simpler version is additive coupling proposed in NICE DDnh et al. 2014), which can be obtained by simply removing the scale item exp s(æ1.a) in affine coupling. Additive coupling layer is volume poeuee-pesrriing and the log-determinant of its Jacobian is always O. Glow improves Real NVP by replacing the fixed shufflingg permutation with x invertible convolution. Since forward and inverse operation of a coupling layer have the same computational efficiency, both likelihood evaluation and sampling(or generation) for coupling flows are equally efficient.
Autoregressive Flow As the building blocks of autoregressive flow, autoregressive transforma- tions model the joint probability p(x) as the product of oie-dimeniiona conditionals:
p(x) = II I(../ae)))
(4)
where the probability of observation n, conditions only on its previous observations Fes- The autoregressive property of autoregressive layer is enforced by specially designed mechanism, e.g. masking.
In PixelCNN (van den Oord et al. 2016), this is implemented masked convolutional layers, which are inherently easier to be parallelized than its counterpart PixeIRNN (Oord et al.] 2016). The likelihood evaluation of PixelCNN takes only one-forward pass, but its inference, i.e. generation, takes (D), since we have to sample pixel-by-pixel. PixeICNN can further parallelized (Reed let al. 2017b) accelerate its inference speed.
3 OTHER RELATED WORKS
Flows for generation Serving powerful decoders, PixeICNN can also be combined with other generative models, e.g. PixelGAN (Makhzani Frey [217] and PixeIVAE (Gulrajani et al. 2016)) Variants of PixelCNN are also used to model audio Van Den Oord et al. 2016). video (Kalchbrenner ctt al., 2017), and text (Kalchbrenner et al., 2016). PixeICNN, combining with attention, is also applied to few-shot autoregressive density estimation Reed al. (017a)) Ho al. (2019) proposes several improvements to coupling flow, reducing its gap to autoregressive flow in terms of density estimation.
Autoregressive models for density estimation Autoregressive models can be specially designed for general-purpose density estimation. Masked Autoencoder for Distribution Estimation (MADE, Germain et al. (2015) is pioneering work that use masked neural networks to model the autore- gressive density. MADE constitutes the building block of two popular normalizing flows: Inverse Autoregressive Flow (IAF, Kingma et al. (2066) and Masked Autoregressive Flow (MAF, Papa- makarios et al. (2017). IAF and MAF are similar but with different computational trade-offs. IAF, providing efficient sampling, is designed to improve the expressiveness the approximate posterior of VAE. MAF is more powerful density estimator which stacks multiple MADES.
3

---

The results in Fig. D and examples in Fig. 2 show that even or 2-pixel left translation could lead obvious increase of Glow's predictive BPDs, while the PixeICNN's predictive BPDs robust to Under review as a conference paper at ICLR 2020
4 METHOD
Let be a flow-based generative model trained on dataset X = {m:} sampled from some un- known p(æ:), and Py (x) be the g's predictive probability density function of sample x.
Semantic-Invare Transformation (SIT) Roughly speaking, SIT can be any transformation that do not change humans' recognition of image semanties. For example, suppose æ dog image. After applying SIT I" to I, T()) is supposed to be high recognized as dog image. As proof of concept evaluation, we limit our evaluations three trivial SITs: image translation, random noise perturbation, and latent factors zeroing (specific to Glow).
Semantic Correlation We probe the correlation between the predictive likelihood of flow y and image semantics by examining the influences of SITs on test samples" likelihoods. Specifically, reasonable observation we should expect is that for SIT (T)))( Py (x) < holds for small positive scalar 8.
report bits-per-dim (BPD), which is given by:
NLL
BPD (h x e) log
(5)
where NLL is the negative log-likelihood of the test sample, h, w. e height, width, number of channels. Lower BPD implies higher likelihood. Throughout this paper, we use BPD and likelihood interchangeably. We refer Supp. A for setup and training details of the target models.
4.1 IMAGE TRANSLATION
Translation invariance is a fundamental property in learning image representations that are robust for downstream tasks. this section, we evaluate the influences of image translations on flows' likelihoods.
Affine
PixelCNN
1.75
1.75
Original
Original
1.50
1p-left
1.50
1p-left
1.25
Zp-left
1.25
2p-left
Mean BPDs
Mean BPDs
1.00
1.00
0.75
0.75
0.50
0.50
0.00
0.00
MNIST Class labels 0-4
MNIST Class labels 0-4
Affine Glow
PixelCNN
3.5
Original
Original
1p-left
1p-left
3.0
3.0
2p-left
2p-left
Mean BPDs
2.5
Mcan gPDs
2.5
2.0
2.0
1.5
1.5
1.0
1.0
0.5
0.0
FashionMNIST Class labels 0-4
FashionMNIST Class labels 0-4
Figure 1: BPDs of (affine) Glow and PixeICNN on MNIST test set with 1-pixel and 2-pixel left translation. Shown are class labels from
See Supp. B. for additional results.

---

Under review as a conference paper at ICLR 2020
pixel translation. This surprising difference can be attributed the difference of their architectures. Glow, like other flows based on coupling transformation layers (Dinh et al. 2016), models the joint probability of the pixels in Noarse-grainge way. They rely multi-scale architecture modeling different level of abstractions in order to achieve competitive BPDs. At higher scale levels, the intermediate tensors have smaller spatial sizes and bigger channel sizes. This is performed the starting point of each scale level with squeeze operation, which trades spatial sizes for channel sizes by transforming h X actually destroys the spatial positions of adjacent pixels, and 1-pixel translation could lead to quite different spatial partitions. While for PixelCNN, the intermediate tensors are not reshaped, and the spatial positions of pixels are kept still; Furthermore, the prediction of each pixel conditions only on a neighborhood of (previous) pixels masked convolution, so translation invariance is preserved.
tensor into a h/2 x w/2 x 4e tensor. Note that squeeze operation
original
1p-left
2p-left
original
1p-left
2p-left
Affine Additive
0.852
1,026
1.118
1.715
1.838
2.083
0.981
1,066
1.132
1.761
1.866
2.007
PixcICNN
0.851
0.849
0.859
1.948
2.030
2.088
2
2
2
Lee
Lee
Lee
Affine 1,306
1.440
1.461
2.967
3.652
3.842
Additive
1,483
1.656
1.701
3.217
3.494
4.252
PixeICNN
1.099
1.104
1.114
2.780
2.810
2.859
Figure 2: Examples from MNIST, FashionMNIST test sets with I-pixel and 2-pixel left translations. Below are predictive BPDs of Glows (Affine and Additive) and PixeICNN.
The foundation of using likelihood -baed models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples Nont than distribution samples Fin, i.e. Pa (Tin) >> Pa ( However, it has been observed that flows assign higher likelihoods on OoD samples than even training samples (Nalisnick et al.) 2018 Choi & Jang. 2018). Analyses of pixel-level datasets statistics in (Nalisnick et al. 2018) show that this may be due to OoD datasets just "sit inside of" in-distribution datasets with roughly the same mean, smaller variance. Surprisingly, similar counter-intuitive likelihood assignment also occurs in in-distribution samples. For example, in Fig. J images with class label 1 are consistently have significantly lower BPDs, i.e. higher likelihoods than samples of other classes. In OoD detection, assume that sample with a higher likelihood indicates that it is more likely to an in-distribution sample. Fol- lowing the same logic, this tells that all images of class label are more likely to be in-distribution samples than samples from other classes, which contradicts the fact that they are all in-distribution samples for sure. We may reasonably suspect that flows' counter-ing intuuer-ntuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics, e.g. different numbers. This kind of counter-intuitive likelihood comparisons exist not only between in-distribution and OoD samples, but also within in-distribution samples from different classes. Sim- ilarly. Theis et al. (2015) find that 1 1-pixel shift of the images could lead to quite different nearest neighbours from the training set, measured in Euclidean distance, which also demonstrate the gap between pixel-level metrics and humans' perception.
Problematic Likelihood Comparisons
4.2 RANDOM NOISE PERTURBATION
Image pixels are discrete integers; and in practice, right amount of real-valued uniform noise is added to dequantize the pixels. For images x E [O..... 255 255/256] scaled to [0, 1]. we usually do find that adding small random perturbations
u. u [0. 1/256] In experiments, out of the coverage the added noise, i.e. u e [0, 1/26/) here, to test images cause flows to give catastrophically higher BPDs.
5

---

Under review as a conference paper at ICLR 2020
perturbed/-
noise
original
masked noise
perturbed/+
4
4
4
Glow
2.102
1.340
2.351
PixeICNN
3.289
1.105
3.277
Glow 2.554
1.986
2.681
PixelCNN
2.972
1.809
2.997
Figure 3: Examples from MNIST, FasHionMNIST test sets perturbed with gaissian noises. The mid- dle column shows the original images. The second and fourth columns are the gaussian noises and masked gaussian noises. The first and fifth columns masks. Below the samples are the BPDs reported by Affine Glow and PixeICNN. Test images are scaled [0, 1]. We could simply generate mask by setting a threshold for the scaled pixels. Here we use m x < 0.3. See Supp. D for more examples.
the perturbed images with noises w/o (+/-)
Humans can robustly recognize semantic objects in images regardless of the backgrounds. So we also evaluate influences of adding random perturbations to only the backgrounds of test images. This can be simply implemented using proper mask:
x f x + e m O noise
(6)
where m the mask, and is small scaling factor ensuring the noise small enough.
In our evaluations, we use unit Gaussian noises, and set the scaling factor = 0.001. Examples in Fig. 3 manifest that adding small noises catastrophically lower samples' likelihoods. Compared to Glow, PixeICNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the influences of the added noise. We get similar results even we keep the semantic objects of test images intact, and add noises only the backgrounds. Note that the Gaussian noises (O, 1) we added are out of the coverage (with < elements) of the uniform noises [0, 1/256] added during training,
theoretically this is expected since models However, it does reveal that flows are not aware of the image semantics, and treat the pixels of objects and the pixels of backgrounds with no discrimination. Other tested noises include e-11/556+ AO 1)) and 1/2256, OJ. and similar results were obtained.
optimized in that areas.
4.3 FREE LIKELIHOOD LOOPHOLE OF GLOW
Algorithm Generate x' by zeroing the latent factors
I: Input: image æ. label y (optional), Glow model g. k L.
2: z + f.oorard((a, y)
Infer the latent factors. Zero the preceding i factors. & Reverse the zeroed latent factors.
3: x= t zero(z,
4: x= + g.reverse(z"..
5: return x
Let us first decompose the Glow architecture into blocks and review their contributions to the final log-likelihood. A Glow consists of a sequence of modules at different scale levels. At each scale level, it starts with squeezze operation, which reshapes the intermediate tensor without contribution to the log-likelihood. Following each squeeze operation is stack of step flow blocks. A step flow block consists of three layers: actnorm, invertible x convolutional layer, and coupling layer. The degedttessss of actnorm and 1x1 convolutional layer input-independern and depend only their inner weights (see Table (Kingma & Dhariwal, 2018)). Additive coupling layer is volume- preserving whose log-determinant O, thus is also input-independent. For affine coupling layer, its

---

Under review as a conference paper at ICLR 2020
log- log-determinant depends the affined half, but is quantitatively small. Compared to Glow with additive coupling layers, affine layers bring only a small improvement of < 0.05 BPD (Kingma & Dhariwal 2018). Then, the intermediary tensor is split into two halves along the channel dimension. One halve is gaussianized with a convolutional block, then the other halve is factored out after being standardized, TTis procedure significantly reduces the amount of computation and memory. We refer to Dinh et al. (2016) for more details due to limited space.
Affine
Additive Glow
1.75
1.75
Original
Original zero <=1 zero ==2
1.50
zero <m1
1.50
1.25
zero <=2
1.25
Mean BPDs
Mean BPDs
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
MNIST Class labels 0-4
MNIST Class labels 0-4
Affine Glow
Additive Glow
3.5
Original
Original
3.0
3.0
zero
2.5
xc=2
2.5
zera <=22
Mean BPDs
Mcan BPDs
2.0
1.5
1.5
0.5
0.5
FashionMNIST Class labels 0-4
FashionMNIST Class labels 0-4
Figure 4: BPDs of affine and additive Glow on MNIST, FashionMNIST test sets by zeroing the preceding 1 (zero z<1) and (zero zς2) latent factors. Shown are classes to 4. See Supp. C for additional results.
So for Glow using additive couplings, the cumulative log-determinant of the flow blocks within a particular scale-level is constant regardless of different samples. Only the log-determinants of gaussianized factorings (between the transitions of different scale-levels) depend on the individual inputs. Denote = { = as the latent variables of input image x. Each z, is the stan- dardized vector of i-th scale level. This simply means that sample æ whose latent variables are close to center 0 will have a higher log- ligelikel,oood. Empirically, it also applies to Glow using affine coupling layers, since the influence of varying the latent variables on the dogeeetessssants of affine coupling layers is quantitatively small compared to its gain.
We could make use of this property to generate samples with higher likelihoods via the invertibility of Glow for free (see Algorithm [ label is optional). We find that the semantic object of a test image depends heavily on the last factored latent žL, rather than the preceding factors. Examples in Fig. 5 show that zeroing the preceding I and 2 latent factors could give samples with obviously lower BPDs but without obvious changes (slightly faded pixel intensities) of the semantic objects. Results in Fig. 4 show that zeroing the first latent factor give the maximum increment of likelihoods.
4.4 IMPLICATIONS ON DISCRIMINATIVE CLASSIFIERS
We also evaluate the influences of these SITs on the performance of discriminative classifiers. In contrast to the obvious change of flows' likelihoods, these small perturbations could decrease the testing accuracies of classifiers to some insignificant, negligible (on MNIST) extent (see Tab I.. The discriminative classifier used here is a shallow residual network of layers on 32 x 32 images in the structure specified in (He et al. 2016).
7

---

Under review as a conference paper at ICLR 2020
original
zero 1<1
zero 252
original
zero #51
zero 2:2
7
7 0.645
7
Glow 0.852
0,588
1.715
1.368
1.356
2
2 1.062
2 1.009
Glow
1.306
2.967
2.526
2.508
Figure 5: Examples of Affine Glow with different zeroed latent variables on MNIST (left) and FashionMNIST (right) test sets. See Supp. C. for results of Additive Glow.
Perturbations
MNIST
FashionMNIST
Clean
99.31%
93.03%
1-pixel
99.12%
89.71%
Gaussian noises
99.31%
93.04%
Zero <=
99.18%
91.44%
Zero <= 2
99.07%
86.22%
Table 1: Test accuracies of discriminative classifiers MNIST, FashionMNIST test sets with dif- ferent perturbations.
Difference to Adversarial Examples Both the SITs use in this paper and adversarial per- turbations are small perturbations, but they are inherently different. Adversarial perturbations are intentionally crafted cause misbehaviors, which are usually specific individual images, and eas- ily fool classifier with almost 0% accuracy. While three SIT's above are universal transformations over all images that take no additional computations and basically come for free.
DISCUSSIONS AND CONCLUSIONS
What is the problem likelihood-based generative models? Discriminative classifiers, trained to extract class-relevant features, are known to be vulnerable to adversarial examples, and give over-confident predictions even for OoD samples. Generative models are supposed to be more robust since they model every pixel information of an image. However, likelihood modeling in high-dimensional space can hard and lead to counter-intuitive observations. was observed that likelihood-batt generative models assign
higher likelihoods OoD samples (alisnick ct al. 2018 Choi Jang 2018). Nalisnick et al. (2018) observe this phenomenon on both flows and VAEs. They decompose the change-of-ver -orr-iable theorem and investigate the influences of different transformation layers, find that the phenomenon still exists regardless whether the transforma- tion polumi--reserving not. Their second-order analysis on pixel statistics suggests that OoD datasets, e.g. MNIST, just sit inside of in-distribution datasets, e.g. FashinMNIST, with roughly the same mean, smaller variance. They suspect that flows may simply fit the pixel intensities without really capture the high-level semantics. Ren (019)) find that the likelihood of an image is mostly dominated by the irrelevant background pixels, and propose remedy correct the original like- lihood with likelihood ratio. Though significantly improves the accuracy of OoD detection, but still fail to answer the question: whether the likelihood ratio shows high correlation to high-level semantics.
This paper differs from previous works and step further to explore the correlations between the likelihood of flow-based generative models and image semantics. Theoretical analyses in (Theis et al. 2015 van den Oord & Dambre 2015) point out an important argument that generative models ability to produce plausible samples is neither sufficiett nor necessary for high likelihood. Results in this paper provide more experimental evidences for this simple argument that even for powerful exact likelihood-baser generative models-flows, the likelihoods of samples be largely weakly
8

---

Under review as a conference paper at ICLR 2020
correlated to the high-level image semantics. Thus, special attention should be paid to this argument before we apply likelihood- based generative models to downstream tasks. For example, considering the weak correlation between flows' likelihoods and image semantics, it may be mappropriate to use them for OoD samples detection. On the other hand, these counter-intuitive behaviours of flows raise our awareness of the gap between the predictive likelihoods of flows and the expectation that these likelihoods can closely relate to the semantics for OoD detection.
What is exactly the likelihood of image? We should keep in mind that the predictive likelihood of a flow is the joint probability of all the image pixels. There is no doubt that flows, trained by maximizing its likelihood, could generate impressive synthesized data. There seem to be no prob- expect that every single generated pixel in a image is the
lem that in terms of image generation, most likely one (hinging on its contextual pixels), However, the likelihood is explicitly modeled on pixels, can be easily influenced by pixel-level modifications. Images' likelihoods significantly decrease even small noises are added to the pixels of backgrounds. For downstream tasks that need some "likelihood" to indicate the object in image is cat, rather than car, the pixels of back- grounds are almost irrelevant. This drive us to think that we may need to model likelihood in some kind of semantic space with some "perceptual" metrics, rather than on raw pixels. One promis- ing direction is to define likelihood of an images on its high-level representation, and successful examples are (Lee 2018 Nilesh A. Ahuja 2019).
REFERENCES
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv: 1,,66449 2015.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arX9:: 1096, 2018
Hyunsun Choi and Eric Jang. Generative ensembles for robust anomaly detection. arXiv preprint 1810.00800 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:/410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605v: 100.00660500803,, 2016.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learming, pp. 881-889, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05011 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00275. 2019.
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099 2016.
Nal Kalchbrenner, Aäron den Oord, Karen Simonyan, Ivo Danihelka. Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proceedings of the 34th Interna- fional Conference on Machine Learning- Dearning-Volume 70, pp. 1771-1779. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Informa- tion Processing Systems, pp. 4743-4751, 2016.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 00236-10245, 2018.
et al Lee, Kimin. A simple unified framework for detecting aut-of-distribution samples and adver- sarial attacks. NIP,, 2018.
Alireza Makhzani and Brendan Frey. Pixelgan autoencoders. In Advances in Neural Information Processing Systems, pp. 1975-1985, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXi::/802.05900 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't know? arXiv preprint vrXXi://810000116,,22 2018.
et al Nilesh A. Abuja. Probabilistic modeling of deep features for outtof-diittibutiin and adversarial detection. arXiv preprini arXiv: 1786, 2019.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 1601.0679, 2016.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in Neural Information Processing Systems 30, 2017.
Scott Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn distributions. arXiv preprini arXiv: 10304, 2017a.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. In Proceedings of the 34th International Conference on Machine Learning- 2921. JMLR. org, 2017b.
Sharning-Volumee Volume 70, pp. 2912-
al Ren, Jie. Likelihood ratios for out-of-distribution detection. arXiv preprint wrXiv:1906.02844 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint a505 055770, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint
2014.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint 1rXiv:1511.01844 2015.
Aäron van den Oord and Joni Dambre. Locally-connected transformations for deep gmms. In International Conference on Machine Learning (ICML): Deep learning Workshop, pp. 1-8, 2015.
Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. SSW, 125, 2016.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Condi- tional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, pp. 4790-4798, 2016.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
10

---

Under review as a conference paper at ICLR 2020
A SETUP OF EXPERIMENTS AND MODELS
The target models Glow and PixelCNN are implemented in Pytorch 1.0, and the implementation of PixeICNN is based proposed in PixenCNN++ are used in our implementation, we still use PixeICNN to annotate our model. We implement both unconditional and conditional versions of Glow. We report the results of conditional Glow in our experiments.
Though several modifications
For Glow, using multi-scale architecture with more levels is critical achieve lower BPDs. We resize the FashionMNIST, MNIST images from 28 x 28 to 32 x 32 and set the number of levels to 5. So the last factored the latent variables will be in code package supplementary material for reproducing the results (We only provide models on MNIST due to the upload size limit). For PixeICNN, both 28 x and 32 versions are provided.
the spatial size. We provide pretrained models
In image translation experiments, report the BPDs of PixelCNN on MNIST images of size 28x 28 the paper. Unlike Glow, PixeICNN treat the pixels as discretized intensity levels, and resizing the image size to 32 x 32 could lead to dangling intensity levels. We also find that if performing
image translation experiment on PixeICNN(32 x 32), 1-pixel or 2-pixel translation will also lead to considerably BPD decreases.
Table 2: Hyperparameters of Glow
Key
Value
Optimizer
Adam
learning rate
0,0002
batch size
64
width of NN module
128
depth (# step flow blocks)
8
n level scale levels)
5
Table 3: Hyperparameters PixeICNN
Key
Value
Optimizer
Adam
learning rate
0.0002
batch size
mixture components of logistic
filters
100
residual blocks per stage

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL RESULTS OF IMAGE TRANSLATION
B.1
ADDITIONAL RESULTS
affine glow
pcnn
1.75
1.75
Original 1p-left 2p-left
Original
1.50
1.50
1p-left
2p-left
Mean BPDs
Mean BPDs
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
MNIST Class labels 5-9
MNIST Class labels 5-9
affine glow
pcnn
3.5
Original
3.5
Original
1p-left
1p-left
2p-left
2p-left
Mean BPDs
2.5
Mean BPDs
2.5
2.0
2.0
0.5
0.5
5
8 Class labels 5-9
8
FashionMNIST
FashionMNIST Class labels 5-9
Figure 6: BPDs of (affine) Glow and PixcICNN on MNIST, FashionMNIST test with different 1-pixel and 2-pixel left translation (Ip-left, 2p-left). Shown are classes from 5-9.
B.2 RESULTS OF ADDITIVE GLOWw
12

---

Under review as a conference paper at ICLR 2020
Additive Glow
additive_glovs
1.75
1.75
Original
Original
1.50
1p-left
1.50
1p-left
Zp-left
1.25
Zp-lert
Mean BPDs
Mean BPDs
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
MNIST Class labels 0-4
MNIST Class labels 5-9
Additive Glow
additive glow
3.5
Original
Original
1p-left
1p-left
2p-left
2p-left
2.5
Mean BPDs
2.5
2.0
2.0
Mean
1.0
1.0
0.5
0.5
0
FashionMNIST Class labels 0-4
FashionMNIST Class labels 5-9
Figure 7: BPDs of (additive) Glow on MNIST and FashionMNIST test sets with different 1-pixel and 2-pixel left translation (Ip-left, 2p-left).
C ADDITIONAL RESULTS OF ZEROING LATENT FACTORS OF GLOW
Affine Glow
Additive Glow
1.75
1.75
Original zero <=1
Original
1.50
1.50
zero =
1.25
zero <=2
1.25
zero x=2
Mean BPDs
Mean BPDs
1.00
0.75
0.75
0.50
0.50
0.00
0.00
6
8 MNIST Class labels 5-9
MNIST Class labels 5-9
Affine Glow
Additive Glow
3.5
3.5
Original
Original
zero <=1
<=1
2.5
zero <=2
2.5
<=2
BPDs
Mean BPDs
2.0
2.0
1.0
10-
0.5
0.5
6
7
9
FashionMNIST Class labels 5-9
FashionMNIST Class labels 5-9
Figure 8: BPDs of affine and additive Glow on MNIST, FaShionMNIST test sets by zeroing the preceding (zero <=1) and (zero <= latent factors.
13

---

Under review as a conference paper at ICLR 2020
C.I
EXAMPLES OF ADDITIVE GLOW
original
zero 1
zero g
original
Zero &<1
ZeTO 2g2
7 0.981
7 0.608
7
Glow
0.493
1.761
1.453
1.432
2
2
2
L
1d
Glow
1.483
1.111
1.013
3.217
2.852
2.772
Glow
0.423
0.071
0.034
1.314
1.068
1.033
Figure 9: Examples of Additive Glow with different zeroed latent variables on MNIST (left) and FashionMNIST (right) test
14

---

Under review as a conference paper at ICLR 2020
D
ADDITIONAL EXAMPLES OF RANDOM NOISE PERTURBATION
perturbed//
noise
original
masked noise
perturbed/+
Glow 2.389 3.413
1.285
2.199
PixelCNN
1.131
3.113
6
6
6
Glow PixelCNN
2.113
1.102
2.161
2.952
1.063
3.180
Figure 10: Examples from MNIST sets perturbed with gaissian noises. The middle column shows the original images. The second and fourth columns are the gaussian noises and masked gaussian noises. The first and fifth columns are the perturbed images with noises w/o (+/-) masks. Below the samples are the BPDs reported by Affine Glow and PixeICNN.
perturbed/-
noise
original
masked noise perturbed/+
Glow
2.966
2.505
2.966
PixeICNN
3.359
2.304
3.153
Glow 2.969
2.274
3,042
PixcICNN
3.202
2.215
3.204
Figure 11: Examples from FasHionMNIST test sets perturbed with gaissian noises. The middle column shows the original images. The second and fourth columns are the gaussian noises and masked gaussian noises. The first and fifth columns the perturbed images with noises w/o (+/-) masks. Below the samples are the BPDs reported by Affine Glow and PixeICNN.
15


Reviews:
[{"review_id": "r1goAJZSKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper studies likelihood-based models of images, such as Glow and PixelCNN. The paper shows empirically that image transformations that preserve semantics (e.g. translations by a few pixels) produce images that have lower probability (density) under such models.\n\nDecision:\n\nThe paper is studying an important topic, which is how to use likelihood-based models correctly, and it warns against the misuse of such models. I agree with the paper's conclusion that we should be careful when using likelihood-based models.\n\nNonetheless, in my opinion the paper is a clear reject. The paper is full of flaws, incorrect statements, poorly constructed arguments, speculative explanations, and superficial descriptions of previous work.\n\nBroadly, the main issue with the paper is the following:\n(a) It begins with flawed assumptions about how a likelihood-based model is expected to behave.\n(b) It tests two likelihood-based models experimentally and finds that they don't behave according to the assumptions.\n(c) It concludes that we need to be cautious when using likelihood-based models.\n\nAs I said, I agree that we should be careful when using likelihood-based models, but I worry that the way the paper reaches this conclusion can mislead and misinform readers. In what follows, I elaborate on specific issues with the paper in more detail.\n\nIssue #1:\n\nThe main flaw of the paper is the assumption that semantic-preserving transformations shouldn't reduce the likelihood of the model (beginning of section 4). This is incorrect. To see why, consider a semantic-preserving transformation x' = T(x). As defined in the paper, a semantic-preserving transformation is one that doesn't change the label y of an image x. This can be formalized as:\n\np(y | x') = p(y | x)\n\nBy Bayes' rule, from the above it follows that:\n\np(x' | y) p(y) / p(x') = p(x | y) p(y) / p(x) \n=> p(x' | y) / p(x') = p(x | y) / p(x)\n\nClearly p(x') can be different from p(x), as long as p(x' | y) is different from p(x | y) by the same factor. Hence, it doesn't follow that if p(y | x') = p(y | x) then p(x') = p(x), which is what the paper incorrectly assumes. To be clear, in the above expressions, p() refers to the true data distribution and not to a model that approximates it.\n\nFor example, consider images of digits, where the digit is generally in the centre of the image. Moving a digit to the corner will result in a less likely image, because it's unlikely that digits appear in corners. However, it won't change the classification of that digit, since all digits are less likely to appear in corners in exactly the same way. In fact, this is exactly why we see in section 4.1 that the likelihood of the model decreases as the image is translated to the left; the model is behaving exactly as it is supposed to.\n\nSimilarly, in section 4.2 where noise is added to the image, the model is again behaving exactly as it is supposed to. Adding Gaussian noise to the image results in a distribution p(x') that is equal to convolving the original distribution p(x) with the noise distribution p(noise) which is an isotropic Gaussian. As a result, p(x') will be a more diffuse version of p(x), hence samples x' will have on average low probability (density) under p(x), exactly as expected, and exactly as the experiment observes.\n\nIssue #2:\n\nThe paper incorrectly assumes that out-of-distribution examples should have low probability (density). This is incorrect, and a common misconception that results from confusing high probability with typicality. In fact, out-of-distribution examples can have high probability (density). Here are two examples that illustrate that:\n\nSuppose you flip a bent coin a million times, with 10% probability of the coin coming up heads. The in-distribution samples (the typical set) are those sequences of coin tosses that have roughly 100 thousand heads. However, the most likely outcome is the all-tails sequence. This outcome is clearly atypical, and many people would agree that it's out-of-distribution, but it has the highest probability.\n\nConsider one million independent Gaussian variables, each with mean 0 and variance 1. Due to the law of large numbers, a typical draw of these variables will have average squared value very close to 1, hence the outcome of all variables being zero is very atypical and many people would agree it's out-of-distribution. However, the all-zero outcome is in fact the one with the highest probability density.\n\nGiven the above, the following two statements copied from the paper are flawed, and potentially misleading:\n\n\"The foundation of using likelihood-based models for OoD detection is that they are supposed to assign much lower likelihoods for OoD samples than in-distribution samples\"\n\"In OoD detection, we assume that a sample with a higher likelihood indicates that it is more likely to be an in-distribution sample\"\n\nFundamentally, I think the issue is that the paper incorrectly assumes that all images with the same semantics (e.g. all images of the digit 3) must be in-distribution. However this is not necessarily true. For example, the true data-generating process of MNIST images (i.e. asking people to write down a digit, scanning it, denoising it, cropping it and centring it) is unlikely to produce images where the digit is not in the centre or the background is noisy. Hence, the images considered in sections 4.1 and 4.2 are indeed out-of-distribution with respect to the true data distribution of MNIST, and are not adversarial examples of the models as the paper suggests.\n\nFurther issues:\n\nThe paper is ostensibly about flow models, but in fact very little is specific to flow models, and most of the discussion, where correct, applies to likelihood-based models in general. In fact, PixelCNN is not a flow model, even though the paper misleadingly describes it as such. PixelCNN can be used to model discrete random variables, whereas flow models are used for continuous random variables (flows for discrete random variables exist, but they are different from PixelCNN). That said, if PixelCNN is used to model continuous random variables then by reparameterization it can be viewed as a flow model with one layer, but that would be an unusual way to present it. Same for WaveNet.\n\n\"It is also believed that flows can be used to detect out-of-distribution(OoD) samples by assigning low likelihoods on them.\"\nBelieved by whom? A citation is needed here.\n\n\"Flows can roughly be divided into two categories [coupling flows and autoregressive flows]\"\nThere are several flows that fall in neither of these categories, such as linear flows, residual flows, planar flows, radial flows, Sylvester flows, neural ODEs, FFJORD, and many others.\n\n\"The autoregressive property of an autoregressive layer is enforced by masking.\"\nThere are other ways of enforcing the autoregressive property (e.g. RNNs); masking is just one of them.\n\nEq. (5) is not correct in general. The bits per dimension should be approximated by:\n\nBPD = (NLL - log|B|) / ((h x w x c) * log2)\n\nwhere |B| is the quantization volume, provided |B| is small. That is, if each pixel with range [0, 1] is quantized into 10 bins, then |B| = 0.1 ^ (h x w x c).\n\n\"This surprising difference can be attributed to the difference of their architectures\" (beginning of page 5)\nThe explanation that follows is speculative, but is not presented as such, which can be misleading.\n\n\"We may reasonably suspect that flows’ counter-intuitive likelihood assignment is dominated by the inherent differences of pixel-level statistics associated to the image semantics\"\nThis is also speculation.\n\n\"PixelCNN is more sensitive to the noises, because its pixel-wise modeling quickly augment and propagate the influences of the added noise\"\nAlso speculation.\n\n\"We find that the semantic object of a test image depends heavily on the last factored latent zL, rather than the preceding factors\"\nAs far as I can see, there isn't evidence in support of that statement in the paper.\n\n\"Considering the weak correlation between flows’ likelihoods and image semantics, it is inappropriate to use them for OoD samples detection\"\nGiven the flawed assumption about the role of image semantics, I don't think there is evidence for that.\n\n\"In terms of image generation, we expect that every single generated pixel in a image is the most likely one\"\nThis is inaccurate; when generating images from a model, we don't get the most likely pixels, but samples from the joint distribution over pixels.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Syx5mzW0YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper raises a problem of the robustness of (log) likelihood computed by invertible flows. The authors show that the changes of likelihood of an image computed by flow-based image generative models have surprisingly weak correlations with semantic changes of image. The flow likelihoods are sensitive to very small changes of pixels that do not affect the semantics of an image. And the likelihoods are less robust against out-of-distribution inputs where we expect strong robustness compared to discriminative models. \nThese claims are validated and supported by several simple experimental results. \n\nThis is an interesting paper: it warns the abuse of likelihoods computed by flow models with several numerical experiments, which are simple but clearly designed to support the claims. \nThese experimental results convince me that the likelihoods of flow-based image generative models are joint distributions of pixel intensities and it is natural such likelihoods are apt to be sensitive in pixel intensity changes, even if they are semantically meaningless. \n\nDesigns of experiments are apparently similar to those of (Nalisnick+, 2018) at the first glance.  I think there is a room to improve the manuscript to clarify the difference from the Nalisnick+’s work. My understanding is that (Nalinsnick+, 2018) is interested in the OOD likelihood behaviors when datasets are swapped, and explains the behaviors of OOK likelihood based on the variance and the curvature of the dataset. This paper directly manipulates the pixel intensities so small that the statistics of the images would not change. \nBTW, the Nalisnick’s paper is accepted and published in ICLR 2019. \n\nIt is a well-known fact that the flow (glow) models can generate natural and high-resolution images by interpolating ``latent hidden vectors’’. This indicates the latent representations are robust against perturbations while pixel intensities are not. So, when this transition of robustness occurs? This seems an interesting problem for me. I’m happy to hear the authors’ opinions about this issue. \n \nSummary\n+ Good research question concerning the robustness of flow models\n+ Simple and understandable claims supported by simple experiments\n+ Easy to read\n- Can be improved more to clarify the difference from the previous work that study the flow model’s likelihoods. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BygO8NNRFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the correlation between likelihood of flow-based generative models and image semantic information, and shows that even small perturbations, like a few pixel translations or noise applied to background, significantly affect models’ likelihoods, which signals that these likelihood models cannot be used for out-of-distribution data detection. However, very similar observations were made in prior works [1] and [2]. In particular, the paper [2] showed that likelihood of PixelCNN is dominated by background pixels which makes the observations in section 4.2 (applying noise to background) unsurprising. The sensitivity of Glow model to even 1-2 pixel translations (section 4.1) and exploiting multi-scale structure of Glow (zeroing latent variables in section 4.3) are interesting, but I believe, not enough for a full paper. Thus, due to the limited novelty, I recommend a weak reject.\n\nOther questions and concerns:\n1. The author claim to introduce “semantic-invariant transformation”. I believe this can be called “data augmentation”, why introduce a new term?\n2. The last bullet point in the introduction is not clearly written.\n3. Equation 1: variable u wasn’t introduced. Paragraph after equation 4: please fix the comma.\n4. The clarity of figure / table captions can be improved, as well as their references in the main text.\n5. The section 4.4 is confusing. Which discriminative classifiers are considered? How are they trained? The Table 1 is not referenced in the main text and the results are not explained or discussed. \n6. The experiments are only performed on MNIST / FashionMNIST datasets. It would help to see experiments on other datasets, e.g. CIFAR-10, SVHN.\n7. Related work section can be elaborated: please, discuss how the observations made in the paper are different from / consistent with [1] and [2].\n\n\n[1] Nalisnick, Eric, et al. \"Do deep generative models know what they don't know?.\" arXiv preprint arXiv:1810.09136 (2018).\n[2] Ren, Jie, et al. \"Likelihood Ratios for Out-of-Distribution Detection.\" arXiv preprint arXiv:1906.02845 (2019).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses the (lack of) correlation between the image semantics and the likelihood assigned by flow-based models, and implications for out-of-distribution (OOD) detection.

The reviewers raised several important questions:
1) precise definition of OOD: definition of semantics vs typicality (cf. definition in Nalisnick et al. 2019 pointed by R1)
There was a nice discussion between authors and the reviewers. At a high level, there was some agreement in the end, but lack of precise definition may cause confusion. I think adding a precise definition will add more clarity and improve the paper.

2) novelty: similar observations have been made in earlier papers cf. Nalisnick et al. 2018. R3 also pointed a recent paper by Ren et al. 2019 which showed that likelihood can be dominated by background pixels. Older work has shown that the likelihood and sample quality are not necessarily correlated. The reviewers appreciate that this paper provides additional evidence, but weren't convinced that the new observations in this paper qualified for a full paper.

3) experiments on more datasets

Overall, while this paper explores an interesting direction, it's not ready for publication as is. I encourage the authors to revise the paper based on the feedback and submit to a different venue.
Decision: Reject