Understanding the Limitations of Conditional Generative Models
Authors: Ethan Fetaya
Joern-Henrik Jacobsen
Will Grathwohl
Richard Zemel
Venue:  (2020)
Keywords: Conditional Generative Models
Generative Classifiers
Robustness
Adversarial Examples

Abstract: Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are a natural choice to solve discriminative tasks in a robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, we investigate robust classification with likelihood-based generative models from a theoretical and practical perspective to investigate if they can deliver on their promises. Our analysis focuses on a spectrum of robustness properties: (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in-distribution inputs. 

Our theoretical result reveals that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers. Experimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CIFAR10. We relate this failure to various undesirable model properties that can be traced to the maximum likelihood training objective. Despite being a common choice in the literature, our results indicate that likelihood-based conditional generative models may are surprisingly ineffective for robust classification.

Published as a conference paper at ICLR 2020
UNDERSTANDING THE LIMITATIONS OF CONDITIONAL GENERATIVE MODELS
Jörn-Henrik Jacobsen" Vector Institute and University of Toronto (ethanf, jjacobs, wgrathwohl, zemel) @ammel@@cal (ccstoronto.edu
Ethan Fetaya"
Will Grathwohl
Richard Zemel
ABSTRACT
Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are natural choice to solve discriminative tasks robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, investigate robust classification with likelihood- tical perspective to investigate if they deliver their promises. Our analysis focuses on a spectrum of robustness properties: (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in-distribution inputs.
generative models from theoretical and prac-
Our theoretical result reveals that it is impossible to guarantee detectability besssriially-peruurbee inputs even for near-optimal generative classifiers. Ex- perimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CI We relate this failure various undesirable model properties that can be traced to the maximum likelihood training objective. Despite being a common choice the literature, our results indicate that likelihood- -based conditional generative models may are surprisingly ineffective for robust classification.
INTRODUCTION
Log- hog-liklihood
-3.1-
-3.2
-3-
2
7
2
4
7
7
7
Figure 1: Linear interpolations of inputs and respective outputs of conditional generative model between two MNIST and CIFAR10 images from different classes. X-axis is interpolation steps and Y-axis negative likelioodooo in bits/dim (higher is more likely under model). MNIST interpolated images are far less likely than real images, whereas for CIFAR10 the opposite is observed, leading to high confidence classification of ambiguous out-of- diitr--diitiitttionn images.
Conditional generative models have recently shown promise to overcome many limitations of their discriminative counterparts. They have been shown to be robust against adversarial attacks (Schott al., 2019 Ghosh al., 2019 [Son al.) 2018 Li al. 2018 Frosst et al.) 2018), enable robust classification in the presence of outliers (Nalisnick et al. 2019b) and to achieve promising results in semi-supervised learning (Kingma et al. 20144] Salimans al. 2016). Motivated by these success stories, we study the properties of conditional generative models in more detail.
Unlike discriminative models, which can ignore class-irrelevant information, conditional generative models cannot discard any information in the input, potentially making it harder to fool them. Further,
"qual contribution

---

Published as a conference paper at ICLR 2020
jointly modeling the input and target distribution should make it easy to detect outtof-distribution inputs. These traits lend hope to the belief that good class-conditiinn aenditionageenerativee important problems faced by discriminative models.
generative models can overcome
In this work, we analyze conditional generative models by assessing them on a spectrum of robustness tasks. (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in- distribution inputs. If a generative classifier is able perform well on all of these, it will naturally be robust to noisy, ambiguous or adversarially perturbed inputs.
Outlier detection in the above settings is substantially different from general out-of-distribution (OOD) detection, where the goal is to use unconditional generative models to detect any OOD input. For the general case, likelihood has been shown to be a poor detector of OOD samples. In fact, often higher likelihood is assigned to OOD data than to the training data itself (Nalisnick et al., 2019a). However, faas--onnitional likelihood necessarily needs to decrease towards the decision -eeusion-boundary for the classifier to work well. Thus, if the class-conditional generative model has high accuracy, rejection of outliers from the wrong class via likelihood may be possible.
Our contributions are:
Provable Robustness We answer: Can we theoretically guarantee that a strong conditional gen- erative model can robustly detect adversarially attacked inputs? In section 2 we show that even a near-perfect conditional generative model cannot be guaranteed to reject adversarially perturbed inputs with high probability.
Assessing the Likelihood Objective We discuss the basis to empirically analyze robustness in practice. We identify several fundamental issues with the maximum likelihood objective typically used to train conditional generative models and discuss whether it is appropriate for detecting ditrri-disttibtiionn inputs.
Understanding Conflicting Results We explore various properties of our trained conditional generative models and how they relate to fact that the model is robust on MNIST but not CIFAR10. We further propose a new dataset where we combine MNIST images with CIFAR background, making the generative task as hard as CIFAR while keeping the discriminative task easy as MNIST, and investigate how it affects robustness.
CONFIDENT MISTAKES CANNOT BE RULED OUT
The most challenging task in robust classification is accurately classifying or detecting adversarial attacks; inputs which have been maliciously perturbed to fool the classifier. In this section we discuss the possibility of guaranteeing robustness to adversarial attacks via conditional generative models.
Detectability of Adversarial Examples In the adversarial spheres work (Gilmer et al. 2018) the authors showed that a model can be fooled without changing the ground-truth probability of the attacked datapoint. This was claimed to show that adversarial examples can lic the data manifold and therefore cannot be detected. While (Gilmer et al., 2018) is an important work for understanding adversarial attacks, it has several limitations with regard to conditional generative models. First, just because the attack does not change the ground-truth likelihood, this does mean the model can not detect the attack. Since the adversary needs to move the input to a location where the model is incorrect, the question arises: what kind of mistake will the model make? If the model assigns low likelihood the correct class without increasing the likelihood of the other classes then the adversarial attack will be detected, as the joint likelihood over all classes moves below the threshold of typical inputs. Second, on the adversarial spheres dataset (Gilmer et al.] 2018) the class supports do not overlap. If we were to train model of the joint density po(x: y) (which does not have 100% classification accuracy) then the KL divergence KL (p(x, v) [po v)), where p(x, y) is the data density, is infinite due to division by zero (note that KL (po (p<, y)) is what is minimized with maximum likelihood). This poses the question, whether small KL (p(x, lpp (z, v)) or small Shannon-Jensee diver- gence is sufficient to guarantee robustness. In the following, we show that this condition is insufficient.
2

---

Published as a conference paper at ICLR 2020
Class Density
Density
Bayes-optimal Classification
Model Classification
A F
Prediction:
Class
Attacks
gix]
A2
Prediction:
Uniformi
Class
Counter example construction. Shown on the left the two class data densities, the right the Bayes-optimal classifier for this problem (assuming Å > >2) and the model we consider. Despite being almost optimal, the model can be fooled with undetectable adversarial examples (red arrows. Detailed description in section
Figure 2:
Why no Robustness Guarantee can be Given The intuition why conditional generative models should be robust is follows: If we have robust discriminative model then the set of confident mistakes, i.e. where the adversarial attacks must reside, has low probability but might be large in volume. For robust conditional generative model, the set of undetectable adversarial attacks, i.e. high-density high-confidence mistakes, has to be small in volume. Since the adversary has to be A close to this small volume set, the A area around this small volume should still be small. This is where the idea breaks down due to the curse of dimensionality. Expanding a set by a small radius can lead to a much larger one even with smoothness assumptions. Based on this insight we build an analytic counter-example for which we can prove that even if
KL (all)) < KL (pllg) <E
(1)
where p = p(z, y) is the data distribution, and q g(x; y) is the model, we can with probability 0.5 take correctly classified input sampled from p. and perturb it by most A to create adversarial example that is classified incorrectly and is not detectable.
We note that the probability in every ball with radius A can be made small desired, excluding degenerate cases. We also assume that the Bayes optimal classifier is confident and is not affected by the attack, i.c. we do not change the underlying class but wrongfully flip the decision of the classifier.
The counter-example goes as follows: Let U (a, b) be the density of a uniform distribution on an annulus in dimension d, {ar Rd S ||ا||| S b} then the data conditional distribution is
p(z|0) (0 1) + (1 A1)
0 o
p(x|1)= = 20
0
0 A2 1
with p(u = 0) ply = 1) = 1/2. Both classes are a mixture of two distributions, uniform on the unit sphere and uniform annulus, as shown in Fig. The model distribution is the following:
U 1 q(x|1)= X2U 1) + (1- (2)
(xx10)=
Le. for y 1 the model is perfect, while for y distribution over the whole domain. If X >> A2 then points in the sphere with radius 1 should be classified as class y = with high likelihood. If X2 >> (1+AJa then the model classifies points in the unit sphere incorrectly with high likelihood. Finally if >> a then almost half the data points will fall in the annulus between 1 and + A and can be adversarially attacked with distance lesser or equal to A by moving them into the unit sphere as seen in Fig. 2 We also note that these attacks cannot be detected as the model likelihood only increases. In high dimensions, almost all the volume of a sphere is in the outer shell, and this can be used to show that high enough dimensions we can get the condition in Eq. [ for any value of e and A (and also the confidence of the mistakes 6). The detailed proof is in the supplementary material.
we replace the mixture with uniform
This counter-example shows that even under very strong conditions, good conditional generative model can be attacked. Therefore no theoretical guarantees can be given in the general case for these models. Our construction, however, does not depend on the learning model but the data geometry. This raises interesting questions concerning the source of the susceptibility to attacks: Is it the model inherent issue with the data?
3

---

Published as a conference paper at ICLR 2020
3
THE MAXIMUM LIKELIHOOD OBJECTIVE
3.1 THE DIFFICULTY IN TRAINING CONDITIONAL GENERATIVE MODELS
Most recent publications on likelihood-based generative models primarily focus quantitative results of unconditonal density estimation (van den Oord et al. 2016 Kingma & Dhariwal 2018
Salimans al. 2017b, Kingma al. 2016; Papamakarios
al. 2017). For conditional density estimation, either only qualitative samples are shown (Kingma & Dhariwal 2018), or it is reported that conditional density estimation does not lead to better likelihoods than unconditional density estimation. In fact, it has been reported that conditional density estimation can lead to slightly worse data likelihoods (Papamakarios et al., 2017 7 Salimans et al., 2017b), which is surprising at first, as extra bits of important information are provided to the model.
Explaining Likelihood Behaviour One way to understand this seemingly contradictory relationship is to consider the objective we use to train our models. When we train a generative model with maximum likelihood (either exactly or through a lower bound) we are minimizing the empirical ap- proximation of g [- log( Po (2. y)]] which is equivalent to minimizing (L((P y) (x, y)). Consider now an image z with a discrete label y, which we are trying to model using Po (z. y). The negative logllikeldood (NLL) objective is:
E [(s.)))[[- (2,,))))
Ep- log
+
log( (o(olll((P((y))))))
(4)
If we model (vllx) with a uniform distribution over classes, then the second term has value of log(C") where C' is the number of classes. This value is negligible compared to the first term Eg-rl- log( (x))] and therefore the "penalty" for completely ignoring class information negligi- ble. So it is not surprising that models with strong generative abilities can have limited discriminative power. What makes matters even worse is that the penalty for confident mis-classification can be unbounded. This may also explain why the conditional ELBO is comparable to the unconditional ELBO (Papamakarios et al. 2017). Another way this can be seen is by thinking of the likelihood as the best lossless compression. When trying encode image, the benefit of the label is at most log(C)) bits which is small compared to the whole image. While these few bits are important for users, from likelihood perspective the difference between the correct p(y (a)) and uniform distribution is negligible. This means that when naively training a class-conditional generative model by minimizing E(xp))] [- ( (x) typically discriminative performance classifier is very poor.
3.2 OUTLIER DETECTION
Another issue arises when models trained with maximum likelihood are used to detect outliers. The main issue is that maximum likelihood, which is equivalent to minimizing KI. (1,, y) Po v)), is known to have "mode-covering" behavior. It has been shown recently in (Nalisnick et al. 2019ab that generative models, trained using maximum likelihood, can be quite poor at detecting dui-of-distribution example. In fact it has been shown that these models can give higher likelihood values, on average, to datasets different from the test dataset that corresponds to the training data. Intuitivily can still hope that high accuracy conditional generative model would recognize an input conditioned on the wrong class as an outlier, as it was successfully trained to separate these classes. In section 4.2 show this is not the case in practice.
While (Nalisnick et al. 2019a) focuses its analysis into dataset variance, we propose this is an inherit issue with the likelihood objective. If it is correct then the way conditional generative models trained is at odds with their desired behaviour. If this is the case, then useful conditional generative model will require a fundamentally different approach.
4 EXPERIMENTS
We now present a set of experiments designed to test the robustness of conditional generative models. All experiments were performed with a flow model where the likelihood can be computed in closed form as the probability of the latent space embedding (the prior) and Jacobian correction term; Sec A.I A.1 for a detailed explanation. Given that we can compute p(æ,8) y) for each class, we can easily compute p(y (r) and classify accordingly. Besides allowing closed-form likelihood computation, the
4

---

Published as a conference paper at ICLR 2020
flexibility in choosing the prior distribution was important to conduct various experiments. In our work we used a version of the GLOW model; details of the models and training is in the supplementary material sec. B We note that the results are not unique to flow models, and verified that similar phenomenon can be seen when training with the PixeICNN+ autoregressive model (Salimans et al. 2017a) in sec.
4.1 TRAINING CONDITIONAL. GENERATIVE MODELS
Here we investigate the ability train conditional generative model with good likelihood and accuracy simultaneously. Usually in flow models the prior distribution in latent space 2 is Gaussian. For classification we used aclass-conditional mixture of 10 Gaussians p(zly)
M( og) We compare three settings: 1) A class-conditional mixture of 10 Gaussians as the prior (Base). 2) A class- conditional mixture of 10 Gaussians trained with an additional classification loss term (Reweighted). 3) Our proposed conditional split prior (Split) described in sec. A.4 in the supplementary material.
Results can be found in table
MNIST
Base
Reweight
Split
CIFAR10
Base
Reweight
Split
% Acc
96.9
99.0
99.3
% Acc
56.8
83.2
84.0
bits/dim
0.95
1.10
1.00
bits/dim
3,47
3.54
3.53
Table I: Comparison between different models.
As we can see, especially on CIFAR 10, pushing up the accuracy to values that are still far from state- of-the-art already results in non-negligible deterioration to the likelihood values. This exemplifies how obtaining strong classification accuracy without harming likelihood estimation is still challenging problem. We note that while the difference between the split prior and re- re-weighted version is not huge, the split prior achieves better NL.. and better accuracy in both experiments. We experimented with various other methods to improve training with limited success, see sec. C in the supplementary material for furture information.
4.2 NEGLIGIBLE IMPACT OF CLASS MEMBERSHIP ON LIKELIHOOD
2000
2000
NLL casss
Bin Count
Bin Count
500
600
600-
400
400
0-
0. 0.7
0.
Negative Log-likel
Negative Log-likelihood
(a) MNIST
(b) CIFARIO
Figure 3: NLL for images conditioned on the correct class vs the highest probability wrong class.
Next we show that even conditional generative models which are strong classifiers do not see images with the corrupted labels as outliers. To understand this phenomenon we first note that if we want the correct class to have a probability of at least 1 - ö then it is enough for the corresponding logit to be larger than all the others by log(C") + log (ه=ف) where C is the number of classes. For C 10 and 5 = le 5 this is about 6, which is negligible relative to the likelihood of the image, which is in the scale of thousands. This means that even for a strong conditional generative model which confidently predicts the correct label, the pair {x,, Vue * y,} (where w is the leading incorrect class) cannot be detected as an outlier according to the joint distribution, as the gap log(p( ((..l11))) much smaller than the variation in likelihood values. In Fig. [ee show this by plotting the histograms of the likelihood conditioned both on the correct class and on the most likely wrong class over the test set. In other words, in order for log(p(<, (Bw)) to be considered outlier the prediction needs to be extremely confident, much more than we expect it to be, considering test classification error.
log(p(a;) (Bm))) is
As we model zero padded datasets as descrihed in section A.4 these numbers are not exactly comparable with the literature.
5

---

Published as a conference paper at ICLR 2020
4.3 ADVERSARIAL ATTACKS AS WORST CASE ANALYSIS
We first evaluate the ability of conditional generative models to detect standard attacks, and then try to detect attacks designed to fool the detector (likelihood function). We evalulate both the gradient based Carlini-Wagner Ly attack (2W- (Carlini & Wagner (017b) and the gradient free boundary attack (Brendel et al., 2018 Results are shown in table on the left. It is interesting to observe the disparity between the CW-L2 attack, which is easily detectable, and the boundary attack which is much harder to detect.
Attacking
Classification
Classification and Detection
MNIST
Reweight
Split
Reweight
Split
CW - Ta Boundary attack
0% (100%%
1% (100%)
17% (100%)
14% (100%)
43% (82%)
36% (80%)
0% (0%)
0% (0%)
CIFAR10
CW-12 T2 Boundary attack
0% (97%)
0% (0%)
6% (99%)
3% (100%) 100% (100%)
67% (100%)
72% (100%)
100% (100%)
Table 2: Comparison of attack detection. Percentage of successful and undetected attacks within L2-distance of E = 1.5 for MNIST and = 33/255 for CIFARIO for proposed models. Number in parentheses is percentage of attacks that successfully fool the classifier, both detected and undetected.
Next we modify our attacks to try to fool the detector well. With the CW-L2 attack we follow the modification suggested in (Carlini & Wagner 2017a) and add an extra loss term (x²) = max(0,- log(p(z')) T where T' is the detection threshold. For the boundary attack turn the C-way classification into a C + 1-way classification by adding another class which is "non-image" and classify any image above the detection threshold as such. We then use a targeted attack to try to fool the network to classify the image into a specific original class. This simple modification to the boundary attack will typically fail because it cannot initialize. The standard attack starts from a random image and all random images are casily detected as "non-image and therefore do not have the right target class. To address this we start from a randomly chosen image from the target class, ensuring the original image is detected as real image from the desired class.
From table (right side) we can see that even after the modification CW-L2 still struggles to fool the detector. The boundary attack, however, succeeds completely on CIFAR 10 and fails completely on MNIST, even when managed to sometimes fool the detector without directly trying. We hypothesize that this is because the area between two images of separate classes, where the boundary attack needs to pass through, is correctly detected as out of distribution only for MNIST and not CIFARIO. We explore this further below.
4.4 AMBIGUOUS INPUTS AS AVERAGE CASE ANALYSIS
To understand why the learned networks are casily attacked on CIFAR but not on MNIST with the modified boundary attack, we explore the probability density of interpolations between two real images. This is inspired by the fact that the boundary attack proceeds along the line between the attacked image and the initial image. The minimum we would expect from a decent generative model is to detect the intermediate middle images as "non-image" with low likelihood. If this was the case and each class was a disconnected high likelihood region, the boundary attack would have a difficult time when starting from a different class image.
Given images To and x from separate classes yo and yı and for a E [0, 1] we generate an intermediate image Za a Ii + (1 a)zo, and run the model various a values to see the model prediction along the line. For endpoints we sample real images that are classified correctly and are above the detection threshold used previously. See Fig. for interpolation examples from MNIST and CIFAR.
In figure (a) we see the average results for MNIST for 1487 randomly selected pairs. As expected, the likelihood goes down moves away from the real images zo and 21. We also the probability of both classes drop rapidly as the network predictions become less confident on the intermediate images. Sampling 100 values uniformly in the range [0, 1] we can also investigate how many of the

---

Published as a conference paper at ICLR 2020
-3.275-
-2
-3.300
-3.325
0.6
0.0
0.2
0.4
0.6
0.8
1.0
(a) Log-likelihood of interpolations on MNIST
(b) log-likeld of interpolations on CIFARIO
4-
Ply-dpr) Po- 103]
0.5
pl- 1143
0.0
0.6
0.8
0.2
0.4
0.8
1.0
(c) Class probability of interpolatioos on MNIST
(d) Class probability interpolations on CIFARIO
Figure 4: Average Log likelihoods and class probabilities for interpolations between data points from different classes, x-axis is interpolation coefficient a. The MNIST model behaves as desired and robustly detects interpolated images. The CIFAR1O model, however, fails strikingly and interpolatd images are consistently more likely than true data under the model.
interpolations all stay above the detection threshold, i.e. all intermediate images are considered real by the model, and find that this happens only in 0.5% of the cases.
On CIFAR images, using 1 1179 pairs, we get very different picture (see fig. 4 (b)). Not only does the intermediate likelihood not drop down, is even higher on average than on the real images albeit to small degree. In classification we also see a very smooth transition between classes, unlike the sharp drop in the MNIST experiment, Lastly, 100% of the interpolated images lay above the detection threshold and none are detected as a "non-image" (for reference the detection threshold has 78.6% recall on real CIFARIO test images). This shows that even with good likelihood and reasonable accuracy, the model still "mashes" the classes together, as one can move from one Gaussian to another without passing through low likelihood regions in-between. also clarifies why the boundary attack is successful on CIFAR but fails completely on MNIST. We note that the basic attack on MNIST is allowed to pass through these low density areas which is why it sometimes succeeds.
4.5 CLASS-UNRELATED ENTROPY TO BLAME
In this section, we show that the difference in performance between CIFAR10 and MNIST can largely be attributed to how the entropy in the datasets is distributed, i.e how much the un- certainty in the data distribution is reduced after conditioning on the class label. For MNIST digits, a large source of uncertainty pixel-space comes from the class label. Given the class, most pixels can be predicted accurately by simply taking the mean of the training set in each class. This is exactly why a linear CIFAR 10,
classifier performs well MNIST. Conversely after conditioning on the class label there still exists consider- able uncertainty. Given the class is "cat," there still exists many complicated sources of uncertainty such as where the cat is and how it is posed. In this dataset, much larger fraction of the uncertainty is not accounted for after conditioning on the label.
9
2
3
This is not a function of the domain or the dimensionality of the dataset, it is a function of the dataset itself.
2
To empirically verify this, we have designed a dataset which replicates the challenges of CIFAR10 10 and places them onto problem of the same discriminative difficulty as MNIST. To achieve this, we simply replaced the black backgrounds of MNIST images with randomly sampled (downsampled and greyscaled) images from CIFAR10. In this dataset, which we call background-MNNS problem is identically predictable from the same of pixels
Figure 5: Top: Samples from
the BG-MNIST-O dataset. Bottom: Samples from conditional genera- tive model trained on the dataset. Note how the model has learnd to capture digit identity.
BNGMMNIS)), the classification
7

---

Published as a conference paper at ICLR 2020
as in standard MNIST but modeling the data density is much
more challenging.
To further control the entropy in fine-gras fine-grained manner, we convolve the background with Gaussian blur filter with various bandwidths to remove varying degrees of high frequency information. With high blur, the task begins to resemble standard MNIST and conditional generative models should perform as they do on MNIST. With low and no blur we expect them to behave as they do on CIFARIO.
Table 3 summarizes the performance of conditional generative models on BG-MNIST. We train models with "Reweighted" discriminative objective as in Section The reweighting allows them to perform well as classifiers but the likelihood of their generative component falls to below CIFARIO levels. More strikingly, now when we interpolate between datapoints we observe behavior identical to our CIFAR10 110 models. This can be seen in Figure 66. Thus, we have created a dataset with the discriminative difficulty of MNIST and the generative difficulty of CIFARIO.
MNIST
BG-MNIST-5
BG-MNIST-1
BG-MNIST-O
CIFAR1O
% Acc
99
99
99
98
84
bits/dim
1.10
1.67
3.30
4.58
3.53
Table 3: Conditional generative models trained on BG-MNIST. BG-MNIST-X indicates the bandwith of blur applied CIFARIO backgrounds.
1.0
-4.8
RKI
0.5
piy-1px)
ply
-5.0-
0.0
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.1
0.6
0.8
1.0
(a) Log- likelihood of interpolations on BG-MNIST-O (b) Class probability of interpolations on BG-MNIST-O
Figure 6: Average log- logllbislsss and class probabilities for interpolations between BG-MNIST-O datapoints. While classification is on par with MNIST models, the likelihood exhibits the same failures as CIFARI0 10 models.
5 RELATED WORK
Despite state of the art performance in many tasks, deep neural networks have been shown to be fragile where small image transformations, (Azulay & Weiss/ 2018) or background object transplant (Rosenfeld al., 2018) can greatly change predictions. In the more challenging case of adversarial pertubations, deep neural networks are known to be vulnerable to adversarial attacks (Akhtar & Mian 2018). and while many attempts have been made train robust models or detect malicious attacks, sigmificant progress towards truly robust models has been made only on MNIST (Schott et al.) 2019 Madry et al. 2017). Even CIFAR10 remains far from being solved from a standpoint of adversarial robustness.
One common belief is that adversarial attacks succeed by moving the data points off the data manifold, and therefore can possibly be detected by generative model which should assign them low likelihood values. Although this view has been challenged (Gilmer et al., 2018), we now discuss how their setting needs to be extended to fully study robustness guarantees of conditional generative models.
Recent work (Song et al. 2018 Frosst et al. 2018 Li et al. 2018) showed that generative model can detect and defend adversarial attacks. However, there is caveat when evaluating detectability of adversarial attacks: the attacker needs to be able to attack the detection algorithm as well. Not doing so has been shown to lead drastically false robustness claims (Carlini & Wagner 2017a). In (Li jet al. 2018) the authors report difficulties training high accuracy conditional generative model on CIFARTO, and resort to evaluation 2-class classification problem derived from CIFAR10 10. While
8

---

Published as a conference paper at ICLR 2020
they do show robustness similar to Carlini- Carlini-Wagner results, they do not apply the boundary attack which we found to break our models on CIFARIO. This highlights the need to utilize a diverse set of attacks. In (Schott et al. 2019) generative model was used not just for adversarial detection but also robust classification on MNIST, leading to state-of-the-ar robust classification accuracy. The method was only shown work MNIST, and is very slow at inference time. However, overall it provides an existence proof that conditional generative models can be very robust in practice. In (Ghohh et al. 2019) the authors also use generative models for detection and classification but only show results with the relatively weak FGSM attack, and on simple datasets. As we see in Fig. aand discuss in section generative models trained on MNIST can display very different behavior than similar models trained on more challenging data like CIFARIO. This shows how success on MNIST may often not translate to success on other datasets.
CONCLUSION
In this work we explored limitations, both in theory and practice, of using conditional generative models to detect adversarial attacks. Most practical issues arise due to likelihood, the standard objective and evaluation metric for generative models by which probabilities can be computed. We conclude that likelihood-bee at odds with one another as important aspects of the problem are not captured by this training and evaluation metric. This has wide-reaching implications for applications like diitttiidittittti detection, adversarial robustness and generalization as well as semi-supervised learning with these models.
density modeling and robust classification may fundamentally be
REFERENCES
Naveed Akhtar and Ajmal S. Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6:14410-14430, 2018.
Aharon Azulay and Yair Weiss. Why do deep convolutional networss generalzze so poorly to small image transformations? CoRR, 2018. URL
12177
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. International Conference Machine Learning, 2019.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Rep- resentations (ICLR), 2018. U http:httts/:/peeeeeetttt
net
net. fwr.m/orummmmo wid=etddddceeeeeee OCWCZ
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017a.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium Security and Privacy, SP, 2017b.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In International Conference on Learning Representations (ICLR), 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In
International Conference on Learning Representations, (ICLR), 2017.
Nicholas Frosst, Sara Sabour, and Geoffrey Hinton. Dareee: Detecting adversaries by reconstruction 2018.
from class conditional capsules. arXiv preprint
Partha Ghosh, Arpan Losalka, and Michael J Black. Resisting adversarial attacks using gaussian mixture variational autoencoders. In Conference on Artificial Intelligence (AAAI), 2019. URL.
https:
www.or///aoo.. wttgg/////// 10..0//180/ 1806 00081
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and lan J. Goodfellow. Adversarial spheres. International Conference on Learning Representations (ICLR), 2018. URL http:http//aai..ccooo wwtp://wwwwwx.p///////x///
https//////////aarrrrrr///////// 02774

---

Published as a conference paper at ICLR 2020
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neu- ral Information Processing Systems 27. 2014. URL http:: wttpp.cccpppp......cccc
5423 wwwpearrtvvevvvvveeee
a -nets.pp..
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Conference on Artificial Intelligence (AAAI), 2018.
Joern-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. In International Conference on Learning Representations, 2019.
URL BtR.hhttps://ppeneeee hwwp:ewwwwwwwweeeeeeeeeee net/ net:/Forumm1c-BkKfgpssst dd=kkfbsscckck
Jöm-Henrik Jacobsen, Arnold W. M. Smeulders, and Fdouard Oyallon. i-revnet: Deep invertible
networks. International Conference Learning Representations (ICLR), 2018. URL http:
07088
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in Neural Information Processing Systems, 10236-10245, 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems (NIPS), pp. 3-81-3589, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743 4151,475,222 2016.
Yingzhen Li, John Bradshaw, and Yash Sharma. Are generative classifiers more robust adversarial attacks? arXiv preprint /802.0652, 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 766006833 2017.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019a. URL /ttL/tttss/////eeeeiiii net
Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. In International Conference on Machine Learning (ICML), 2019b.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), 2015.
Amir Rosenfeld, Richard S. Zemel, and John K. Tsotsos. The elephant in the room. CoRR, 2018. URL hRR/hht:://axxix org/ wttgg//bb/888.... 03305
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, pp. 2234,224,, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: A pixelenn implementation with discretized logistic mixture likelihood and other modifications. In ICLR, 2017a.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 1711001701.05517. 2017b.
10

---

Published as a conference paper at ICLR 2020
L. Schott, J. Rauber, W. Brendel, and M. Bethge. Towards the first adversarially robust neural network model on mnist. 2019. TRL https::////// / ott../9.//..................
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Lever- aging generative models to understand and defend against adversarial examples. In International Conference on Learning Representations (ICLR), 2018.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances Neural Information Processing Systems, pp. 4790-4798, 2016.

---

Published as a conference paper at ICLR 2020
A TRAINING CONDITIONAL GENERATIVE MODELS
A.1 LIKELIIOOD-BASED GENERATIVE MODELS GENERATIVE CLASSIFIERS
We present brief overview of flow-based deep generative models, conditional generative models, and their applications to ad adversarial example detection.
Flow based generative models Rezende Mohamed (015); Dinh et al. (2015 2017): Kingma & Dhariwal 2018) compute exact densities for complex distributions using the change of variable formula. They achieve strong empirical results Kingma & Dhariwal (2018) and the closed form likelihood makes them easier to analyze than the closely related VAE Kingma et al. (2014). The main idea behind flow-based generative models is to model the data distribution using a series of bijective mappings NN = f(x) JN f' fi (x)) where NN has known simple distribution, e.g. Gaussian, and all fa are parametric functions for which the determinant of the Jacobian can be computed efficiently. Using the change of variable formula have log (g(((x)) log(p(=x))
ΣN log(1 det (c)()))) where J; is the Jacobian of ft and 2₁-1 = fi-1... fi (z).
The standard way to parameterize such functions fü is by splitting the input 2 into two 2;-1 (=|_1:2(_1) and chose
fc 1 s(2)_ of t(LL))
(5)
which is invertible as long as s s( = 8 + 0 and we have log([ (et(() ()) Σ, log(Is;())
For images the splitting is normally done in the channel dimension. These models are then trained by maximizing the empirical log likelihood (MLE).
straightforward way to turn this generative model into a conditional generative model is to make p(zN) a Gaussian mixture model (GMM) with one Gaussian per class, i.e. p(ZN y) N (py. Eg). Assuming p(y) is known, then maximizing log(p(x,
is equivalent to maximizing log(p( (z)
log(p(zN (y)) + > log( det (2)))) At inference time, one can classify by simply using Bayes rule. Note that directly optimizing log(p(xlv)) results in poor classification accuracy as discussed in section 3 This issue was also addressed in the recent hybrid model work Nalisnick et al. (2019a).
We will now describe some of the various approaches we investigated in order to train the best possible flow-based conditional generative models, to achieve better trade-off betwen classification accuracy and data-likelihood as compared to commonly-used approaches. We also discuss some failed approaches in the appendix.
A.2 REWEIGHTING
The most basic approach, which has been used before in various works, is to reweight the discrimi- native part in eq. (4). While this can produce good accuracy, it can have an unfavorable trade-off with the NLL where good accuracy comes with severely sub-optimal NL... This tradeoff has also been shown in Nalisnick et al. (2019a) where they train a somewhat similar model but classify with a generalized linear model instead of a Gaussian mixture model.
A.3 ARCHITECTURE CHANGE
Padding channels has been shown to increase accuracy in invertible networks Jacobsen al. (2018)
Behrmann et al. (2019). This helps ameliorate a basic limitation in bijective mappings (see Eq. (5). by allowing to increase the number of channels as a pre-prccessing step. Unlike the discriminative i-RevNet, we cannot just pad zeros as that would not be continuous density. Instead we pad channels with uniform(0, random noise. In effect we do not model MNIST and CIFAR 10 as is typically done in the literature, but rather the zero-padded version of those. While the ground-truth likelihoods for the padded and un-padded datapoints are the same due to independence of the uniform noise and unit density of the noise, this is not guaranteed to be captured by the model, making likelihoods very similar but not exactly comparable with the literature. This is not an issue for us, as we only compare models on the padded datasets.
12

---

Published as a conference paper at ICLR 2020
A.4 SPLIT PRIOR
One reason MLE is bad at capturing the label is because a small number of dimensions have small effect on the NLL. Fortunately, of the conditional class information negligible for the data log likelihood, we choose to model it its distinct subspace, as proposed by [Jacobsen et al. (2019). Thus, we partition the hidden dimensions 2 = (Ze, 2n) and only try to enforce the low-dimensions 2, to be the logits. This has two advantages: 1) we do not enforce classccondiiional dimensions to be factorial; and 2) we can explicitly up up-weight the loss on this subspace and treat it as standard logits of a discriminative model. similar approach is also used by semi-supervised VAEs Kingma et al. (2014). This lets us jointly optimize the data log-likelln ge-likkllihod alongside classification objective without requiring most of the dimensions to be discriminative. Using the factorization p(2,. 2n B) = p(=+ly) p(2, 2g. y) we model p(zely) Gaussian with class conditional mean E; (0, .... 0, .,. and covariance matrix scaled by a constant. The distribution p(z, 2z. y) is modeled as a Gaussian where the mean and variances function of y and Zn-
this property to advantage. As the contribution
B IMPLEMENTATION DETAILS
We pad MNIST with zeros so both datasets are 32x32 and subtract 0.5 from both datasets to have [-0.5,0.5] range. For data augmentation we do pytorch's 'ss random crop with a padding of and "edge' padding mode, and random horizontal flip for CIFARIO only.
The model is based on GLOW with levels, affine coupling layers. 1x1 convolution permutations and actnorm in a multi-scale architecture. We choose 128 channels and 12 blocks per level for MNIST and 256 channels and 16 blocks for CIFARIO. In both MNIST and CIFARIO experiments the number of channels with uniforrm(0,) noise which we scale down to the range [0, 2/256] (taking it into account in the Jacobian term). One major difference is that do the squeeze operation the end of each level instead of the beginning, which is what allows us to use 4 levels. This is possible because with the added channels the number of channels is even and the standard splitting is possible before the squeeze operation.
double
The models are optimized using Adam, for 150 epochs. The initial learning rate is 1e 3, decayed by a factor of 10 every 60 epochs. For the reweighted optimization the objective
loss = - log(p(>lv))/ D log(p(y(z))
(6)
where D is the data dimension (3x32x32 for CIFAR, 1x32x32 for MNIST).
For adversarial detection we use threshold of 1.4 for MNIST (100% of test data are below the threshold) and 4. for CIFAR10 (78.6% of test images are below the threshold).
C NEGATIVE RESULTS
In this work explored many ideas in order to achieve better tradeoff between accuracy with little or impact.
C. ROBUST PRIORS
Since the Gaussian prior is very sensitive to outliers, one idea was that confident miss- chaisssclassiiaaiionns carry strong penalty which might result in "messing" all the classes together. A solution would be to replace the Gaussian with more robust prior, e.g. Laplace or Cauchy. Another idea we explored is mixture of Gaussian and Laplace or Cauchy using the same location parameter. our experiments we did not any significant difference from the Gaussian prior.
C.2 LABEL SMOOTHING
Another approach to try to address the same issue is a version of label smoothing. In this new model the Gaussian clusters latent variable that is equal to the real label with probability
13

---

Published as a conference paper at ICLR 2020
and uniform the other labels with probability Using this will bound the enror for confident clsssalaasiiication as long as the data close to one of the Gaussian centers.
C.3 FLOW-GAN
As claimed the main issue is with the MLE objective, it seems like better objective is to optimize KL (p(x,,)) y)]ppp(,, y)) or the -haennnhnnnon divergence as this KL term highly penalized for muis-cllasifiiattinn It is also more natural when considering robustness against adversarial attacks. Optimizing this directly is hard, but generative adversarial networks (GANs) Goodfelloo et al. (2014) in theory should also optimize this objective. Simply training GAN would not work interested in the likelihood value for adversarial detection and GANs only let you sample and does not give you any information regarding an input image.
Since flow algorithms are bijective, we could combine the two objective as was done in the Bow-GAN paper Grover et al. (2018). We trained this approach with various conditional-GAN. alternatives and found it very hard train. GANs are know be unstable train, and combining them with the unstable flow generator is problematic.
D ANALYTICAL COUNTER EXAMPLE:
Assume ply = 1) ply = 0) = glv = 1) =) 1/2 and
p(z[0)
(7)
p(z[1) (1-)))))+1)))0)()))))))
(8)
(x(0)) = U(),++A)
(9)
(x111) (1-
(10)
where U(a, b) is the uniform distribution on the annulus (a b) dimension d.
for Rd a || s b} in
Lemma 1. For ||>||| < we have
A1 p(0/z) \ + A2
(11)
(00)z) = 1+ d
(12)
Proof. The Ua, b) density (when it isn't 't zero) is caiba-aa) where ca is the volume of the d- dimensional unit ball. The proof follows by simple use of Bayes rule.
(I+A)a we can have the model switch wrongfully predictions from y to y = when we move x from the annulus R R1, 1 + A) to (0,
so by having A1 >> X2 >>
I
Lemma 2. If A > (ItAfa and \ < -1-e- then KL(g(æ, v) P(r, w)) Se
Proof. Using the chain rule for KL divergence, (L(P(a,,))
kL(((W)Il(v))
E,, (P((cl))))Q(((( we get that KL(g(æ, v)l v)) = KL(g(=ly = P) = 0)). We now have
- Ca(1 A)d
1
CalI+Af
=
log
A C
(13)
* Ca(1 + A)d
Ca(1+AF
log(A> + A)4)
log
(14)
I-
(1+ A)d
Ca(1I+A/'-1)
()+ -1 log
(1+A)) (1 A)
< log
<e
(15)
A1) A)d
1 1- X1
14

---

Published as a conference paper at ICLR 2020
Lemma 3. 1f X
1 and X1 dlog(1+A) then v)) sc
(1-A)a
Proof. Again using the KL chain rule we have
/
1 log
KL(P(x = 0)1l((1ly= =
A1
Jn Cd
(16)
Ca(I+Aaa
(1-A1) Cal(1+A)d-1)
IM Cal(1 + A) -)
(1 A1)
log
A)dlog(1 + + <E
(17)
Ca(I+A
Proposition 1. For all (c, A)
there is a distribution and an approximation q in dimension log( +lgg() such that log(1+A)
d=0
[L(g(x, f)lp,(, v)) se KL(p(z, V)ll((2 y)) <e
(18)
but with probability greater then 1/3 over samples a ~ there is an adversarial example I satisfying:
1. M1(x) yp(a) with (x))())))) and q(yg())aa) greater or equal 1- 8. The original point is classifier correctly and confidently.
2. (a(x) y()). Vo (3) (.)). We change the prediction without changing the ground-truth label.
3. 4(y, (z) P(Up ④ x)) > 1 - 5. The classifier is confident in its wrong prediction.
4. ||2 - A. We make a small change to the inputs.
5. The density (()) is greater or equal to the median density, making the attack undetectable by observing (xx).
6. For A < 1 the probability in any radius ball can be made as small as desired.
7. The total variation of the distribution can be made as small as desired.
The last two conditions exclude degenerate trivial counter-exmaples, one where the whole distribution support is in a A radius ball and does indeed represent small pertubation. The other condition excludes "pathological" distributions ,e.g. misclassification on a dense zero measure set like the rationals.
Proof. In order to satisfy conditions 1-5, using previous lemmas, it enough that
A - ArtAa
2. 1+)2(1+A)2 5
3. A < 11-60 -
4. A (T+AF
5. A dlog(i+A)
By setting X - we can easily satisfy condition 1. It is not hard to see that condition 2 is
equivalent to \ > (_6)
(I+AF which superseeds condition when d 1/2. Condition can be
15

---

Published as a conference paper at ICLR 2020
satisfied with A1 c/2 by using 1 e for 1/2.
This boils down to ensuring d is large enough that there is a valid X such as
< A < dllog(1 + A)
(19)
Ā (1+A)
Which true for large enough as the l.h.s decays exponentially while the r.h.s linearly.
Condition 6 is trivial as the radius of the support is fixed so as long as A < the probability in any A radius ball decays exponentially. Regarding total variation, we note that from the divergence theorem this can be bounded by term that depends the surface area of shperes with fixed radius which decreases to zero as goes to infinity.
E PIXELCNN+
We trained a conditional PixcICNN++ where instead of predicting each new pixel using mixture of 10 components, we use one mixture component per class. Using reweighting we train using the following objective -log(p(a (zll)))/dl (ely))/iim trainig, i.e. a O, results in very poor accuracy, while reweighting the classification score results in much better accuracy but worse NLL.
llog(1 (p(y[=)). As one can see from table 4 standard
Q
acc (%)
bits/dim
0
25.48
3,05
1000
85.78
3.34
Table Accuracy and NLL for PixelCNN+ on CIFARIO
16


Reviews:
[{"review_id": "rJx8deVBcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "paper summary:\nThe authors claim that likelihood based generative models are not as robust to noise as general consensus claims them to be. To prove this authors make use of adversarial, ambiguous and incorrectly labeled in distribution inputs. Authors address issues regarding robustness in near perfect conditional generative models as well as assess the robustness of the likelihood objective.\n\nPros of the paper:\n1) Authors make well motivated arguments about how a near perfect generative model is also susceptible to attacks by providing examples that are adversarial, and have high likelihood and yet are incorrectly labeled.\n2) They also demonstrate how class conditional generative models have poor discriminative power.\n\nCons:\n1) The experiments section is written very poorly. This section relies heavily on the supplement making it hard to read due to the constant back and forth between the results and details of the experiments.\n2) Experiments seem largely limited. Comparisons on image  data sets such as MNIST and CIFAR10 alone are not convincing enough to establish generalizability of the proposed theory. For example, the hypothesis could completely fail on text based generative models.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Sklh8s_atB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank the authors' for their response, and have read the other reviews.\n\nThis paper demonstrates some theoretical and practical limitations on the use of likelihood based generative models for detecting adversarial examples. They construct a simple counterexample showing that there are adversarial examples for an arbitrarily accurate model (as measured by KL) that are not detectable by diminished likelihood of the model (as the dimension increases). Extending the work of Gilmer et al, this proves that there can be no general robustness guarantee for conditional generative models (Bayes classifiers). They provide compelling empirical evidence that while conditional normalizing flows trained on MNIST can be effective in detecting and defending adversarial attacks, these models trained on CIFAR10 are not. Surprisingly, it is shown that linear interpolations between images of different classes yield higher likelihoods for the CIFAR10 models and that class has little impact on model likelihoods. This goes some way in explaining why the detection is not effective on CIFAR, but questions still remain.\n\nThe paper makes fairly modest claims, but does a good job at demonstrating them and shedding some light on the issue. The experiments are thorough and fit into a growing body of evidence that the likelihoods of normalizing flows and other image based likelihood models may not be that informative or well calibrated, where past work has focused on out-of-distribution detection. My only major complaint with the paper is that it is not clear to what extent the theoretical and practical problems are related. As mentioned in the paper, the counterexample construction depends on the geometry of the data rather than the learning model. It could be that for both the MNIST and CIFAR10 datasets, the geometry is such that robustness garauntees are possible, and that the discrepancy in detection and interpolation arises because the normalizing flow has modeled the MNIST distribution much better than the CIFAR10 distribution. In this case we might hope that using conditional likelihood models for adversarial detection can be made effective, but that effort needs to be placed into improving the modeling capability. It's not obvious how to probe this distinction, but it would be good if this was given some thought in the paper. Also it would be good to see the attack detection numbers on BG-MNIST.\n\n\nComments:\nDifficulty in training conditional generative models:\nI believe in the two papers you cite the models do not use the label as input, but rather there is a separate model for each class? The overfitting is likely why the models had slightly lower conditional likelihood. As an aside, there are a couple of other examples of conditional normalizing flow models on images that use a mixture of Gaussians in the latent space [1], [2].\n\neq. 4: In the paper it is said that the second term in eq 4 is at most log(C), because the uniform distribution would have this value and that therefore this is negligibly small in comparison to the other term. Why exactly is this the case, couldn't the data entropy term be smaller in principle even if it's larger in practice? Or is the argument that the data entropy term scales with the dimensionality, but the label term does not leading to an imbalance? This could use some clarification.\n\nA3: What is meant by ‘While the ground-truth likelihoods for the padded and un-padded datapoints are the same due to independence of the uniform noise and unit density of the noise’ in the appendix section A3? Wouldn't the ground truth negative log likelihoods would increase by the entropy of the uniform noise? Also, then in the bits per dimension calculations is the dimension the number of unpadded dimensions or the padded dimensions?\n\n\n[1] Izmailov, Pavel, et al. \"Semi-Supervised Learning with Normalizing Flows.\" Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019.\n[2] Atanov, Andrei, et al. \"Semi-Conditional Normalizing Flows for Semi-Supervised Learning.\" Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1ljBraStH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post rebuttal:\n\nThank you for your response. I appreciate the authors add an experiment on BG-MNIST, which shows the intermediate trend of MNIST and CIFAR-10.\n\nAs the authors mentioned, the reweighting scheme could be a simple yet effective way to address the problem of current likelihood-based models. While there is room for improvement to further develop the method, the current version of the paper would be a good contribution to the community.\n\nHence, I raise my score from 3 to 6.\n\n----------------------------------------\n\nSummary:\nThis paper investigates some limitations of the conditional generative models (or generative classifiers). First, the authors present a counter-example that a good generative classifier fails to detect adversarial attacks. Second, the authors claim that the marginal and conditional terms of the likelihood objective are the source of the problem. Finally, the authors demonstrate some experiments on adversarial attacks, out-of-distribution (OOD) samples, and noisy labels.\n\nPros:\n- While generative classifiers are believed to be more robust than the discriminative counterparts [1], the authors present a counter-example that it may not be true.\n- The authors investigate the marginal and conditional terms of the likelihood objective and demonstrate empirical results that the model fails to capture the outliers.\n\nCons:\n\n1. The imbalance issue of the likelihood objective is not surprising.\n\nAs the data x is far complex than the class y, it is expectable that the penalty from modeling p(x) is larger than the penalty from classifying p(y|x). As mentioned in Table 1 and Appendix A.2, balancing two terms indeed improves the classification performance. However, to meet the high standard of ICLR, the authors should propose an alternative or modification of the likelihood which resolves the existing limitations. For example, [2] decomposes the semantic and background parts to improve the OOD detection using likelihood models.\n\n2. The experiments are not extensively studied.\n\nThe authors conduct experiments on two datasets: MNIST and CIFAR-10. The authors may present more results on other datasets (e.g., SVHN or CIFAR-100) and convince if their findings are consistent. Also, some observations seem to be an inheritance of the datasets, e.g., Figure 4 is natural since MNIST has disjoint support and CIFAR-10 has a continuous one.\n\nMinor comments:\n- On page 8, ',' should be moved after (Azulay & Weiss, 2018).\n- On page 8, (Schott et al.) should be changed to '\\citet' format.\n- PixelCNN++ is doubly cited.\n\n\n[1] Li et al. Are Generative Classifiers More Robust to Adversarial Attacks? ICML 2019.\n[2] Ren et al. Likelihood Ratios for Out-of-Distribution Detection. NeurIPS 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents theoretical results showing the conditional generative models cannot be robust. The paper also provide counter examples and some empirical evidence showing that the theory is reflected in practice. Some reviewers doubt how much of the theory holds in reality, but still they think that this paper could be a useful for the community. After the rebuttal period, R2 increased their score and it seems that with the current score the paper can be accepted.
Decision: Accept (Poster)