Title: UNDERSTANDING THE LIMITATIONS OF CONDITIONAL GENERATIVE MODELS

Abstract: Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are a natural choice to solve discriminative tasks in a robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, we investigate robust classification with likelihood-based generative models from a theoretical and practical perspective to investigate if they can deliver on their promises. Our analysis focuses on a spectrum of robustness properties: (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in-distribution inputs. Our theoretical result reveals that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers. Experimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CIFAR10. We relate this failure to various undesirable model properties that can be traced to the maximum likelihood training objective. Despite being a common choice in the literature, our results indicate that likelihood-based conditional generative models may are surprisingly ineffective for robust classification.

1. INTRODUCTION:

Figure 1 : Linear interpolations of inputs and respective outputs of a conditional generative model between two MNIST and CIFAR10 images from different classes. X-axis is interpolation steps and Y-axis negative log-likelihood in bits/dim (higher is more likely under model). MNIST interpolated images are far less likely than real images, whereas for CIFAR10 the opposite is observed, leading to high confidence classification of ambiguous out-of-distribution images.

Conditional generative models have recently shown promise to overcome many limitations of their discriminative counterparts. They have been shown to be robust against adversarial attacks (Schott et al., 2019; Ghosh et al., 2019; Song et al., 2018; Li et al., 2018; Frosst et al., 2018) , to enable robust classification in the presence of outliers (Nalisnick et al., 2019b) and to achieve promising results in semi-supervised learning (Kingma et al., 2014; Salimans et al., 2016) . Motivated by these success stories, we study the properties of conditional generative models in more detail.

Unlike discriminative models, which can ignore class-irrelevant information, conditional generative models cannot discard any information in the input, potentially making it harder to fool them. Further, jointly modeling the input and target distribution should make it easy to detect out-of-distribution inputs. These traits lend hope to the belief that good class-conditional generative models can overcome important problems faced by discriminative models.

In this work, we analyze conditional generative models by assessing them on a spectrum of robustness tasks. (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled indistribution inputs. If a generative classifier is able to perform well on all of these, it will naturally be robust to noisy, ambiguous or adversarially perturbed inputs.

Outlier detection in the above settings is substantially different from general out-of-distribution (OOD) detection, where the goal is to use unconditional generative models to detect any OOD input. For the general case, likelihood has been shown to be a poor detector of OOD samples. In fact, often higher likelihood is assigned to OOD data than to the training data itself (Nalisnick et al., 2019a) . However, class-conditional likelihood necessarily needs to decrease towards the decision-boundary for the classifier to work well. Thus, if the class-conditional generative model has high accuracy, rejection of outliers from the wrong class via likelihood may be possible.

Our contributions are::

Provable Robustness We answer: Can we theoretically guarantee that a strong conditional generative model can robustly detect adversarially attacked inputs? In section 2 we show that even a near-perfect conditional generative model cannot be guaranteed to reject adversarially perturbed inputs with high probability.

Assessing the Likelihood Objective We discuss the basis to empirically analyze robustness in practice. We identify several fundamental issues with the maximum likelihood objective typically used to train conditional generative models and discuss whether it is appropriate for detecting out-of-distribution inputs.

Understanding Conflicting Results:

We explore various properties of our trained conditional generative models and how they relate to fact that the model is robust on MNIST but not on CIFAR10. We further propose a new dataset where we combine MNIST images with CIFAR background, making the generative task as hard as CIFAR while keeping the discriminative task as easy as MNIST, and investigate how it affects robustness.

2. CONFIDENT MISTAKES CANNOT BE RULED OUT:

The most challenging task in robust classification is accurately classifying or detecting adversarial attacks; inputs which have been maliciously perturbed to fool the classifier. In this section we discuss the possibility of guaranteeing robustness to adversarial attacks via conditional generative models.

Detectability of Adversarial Examples:

In the adversarial spheres work (Gilmer et al., 2018) the authors showed that a model can be fooled without changing the ground-truth probability of the attacked datapoint. This was claimed to show that adversarial examples can lie on the data manifold and therefore cannot be detected. While (Gilmer et al., 2018) is an important work for understanding adversarial attacks, it has several limitations with regard to conditional generative models. First, just because the attack does not change the ground-truth likelihood, this does not mean the model can not detect the attack. Since the adversary needs to move the input to a location where the model is incorrect, the question arises: what kind of mistake will the model make? If the model assigns low likelihood to the correct class without increasing the likelihood of the other classes then the adversarial attack will be detected, as the joint likelihood over all classes moves below the threshold of typical inputs. Second, on the adversarial spheres dataset (Gilmer et al., 2018) the class supports do not overlap. If we were to train a model of the joint density p θ (x, y) (which does not have 100% classification accuracy) then the KL divergence KL (p(x, y)||p θ (x, y)), where p(x, y) is the data density, is infinite due to division by zero (note that KL (p θ (x, y)||p(x, y)) is what is minimized with maximum likelihood). This poses the question, whether small KL (p(x, y)||p θ (x, y)) or small Shannon-Jensen divergence is sufficient to guarantee robustness. In the following, we show that this condition is insufficient.

Δ:

p(x | 0) = 1 -λ 1 p(x | 0) = λ 1 Class 1 Density Class 0 Density p(x | 1) = λ 2 p(x | 1) = 1 -λ 2 q(x | 1) = λ 2 q(x | 1) = 1 -λ 2

Bayes-optimal Classification:

Model Classification Prediction: Class 0 Why no Robustness Guarantee can be Given The intuition why conditional generative models should be robust is as follows: If we have a robust discriminative model then the set of confident mistakes, i.e. where the adversarial attacks must reside, has low probability but might be large in volume. For a robust conditional generative model, the set of undetectable adversarial attacks, i.e. high-density high-confidence mistakes, has to be small in volume. Since the adversary has to be ∆ close to this small volume set, the ∆ area around this small volume set should still be small. This is where the idea breaks down due to the curse of dimensionality. Expanding a set by a small radius can lead to a much larger one even with smoothness assumptions. Based on this insight we build an analytic counter-example for which we can prove that even if

Attacks:

Prediction: Class 1 q(x | 0) = ∪ ) Uniform(

EQUATION

where p = p(x, y) is the data distribution, and q = q(x, y) is the model, we can with probability ≈ 0.5 take a correctly classified input sampled from p, and perturb it by at most ∆ to create an adversarial example that is classified incorrectly and is not detectable.

We note that the probability in every ball with radius ∆ can be made as small as desired, excluding degenerate cases. We also assume that the Bayes optimal classifier is confident and is not affected by the attack, i.e. we do not change the underlying class but wrongfully flip the decision of the classifier.

The counter-example goes as follows: Let U (a, b) be the density of a uniform distribution on an annulus in dimension d, {x ∈ R d : a ≤ ||x|| ≤ b} then the data conditional distribution is

p(x|0) = λ 1 U (0, 1) + (1 -λ 1 )U (1, 1 + ∆) 0 ≤ λ 1 ≤ 1 (2) p(x|1) = λ 2 U (0, 1) + (1 -λ 2 )U (2, 3) 0 ≤ λ 2 ≤ 1

with p(y = 0) = p(y = 1) = 1/2. Both classes are a mixture of two distributions, uniform on the unit sphere and uniform on an annulus, as shown in Fig. 2 . The model distribution is the following:

q(x|0) = U (0, 1 + ∆) (3) q(x|1) = λ 2 U (0, 1) + (1 -λ 2 )U (2, 3)

i.e. for y = 1 the model is perfect, while for y = 0 we replace the mixture with uniform distribution over the whole domain. If λ 1 λ 2 then points in the sphere with radius 1 should be classified as class y = 0 with high likelihood. If λ 2 >> 1 (1+∆) d then the model classifies points in the unit sphere incorrectly with high likelihood. Finally if 1 >> λ 1 then almost half the data points will fall in the annulus between 1 and 1 + ∆ and can be adversarially attacked with distance lesser or equal to ∆ by moving them into the unit sphere as seen in Fig. 2 . We also note that these attacks cannot be detected as the model likelihood only increases. In high dimensions, almost all the volume of a sphere is in the outer shell, and this can be used to show that in high enough dimensions we can get the condition in Eq. 1 for any value of and ∆ (and also the confidence of the mistakes δ). The detailed proof is in the supplementary material. This counter-example shows that even under very strong conditions, a good conditional generative model can be attacked. Therefore no theoretical guarantees can be given in the general case for these models. Our construction, however, does not depend on the learning model but on the data geometry. This raises interesting questions concerning the source of the susceptibility to attacks: Is it the model or an inherent issue with the data?

3. THE MAXIMUM LIKELIHOOD OBJECTIVE:



3.1. THE DIFFICULTY IN TRAINING CONDITIONAL GENERATIVE MODELS:

Most recent publications on likelihood-based generative models primarily focus on quantitative results of unconditonal density estimation (van den Oord et al., 2016; Kingma & Dhariwal, 2018; Salimans et al., 2017b; Kingma et al., 2016; Papamakarios et al., 2017) . For conditional density estimation, either only qualitative samples are shown (Kingma & Dhariwal, 2018) , or it is reported that conditional density estimation does not lead to better likelihoods than unconditional density estimation. In fact, it has been reported that conditional density estimation can lead to slightly worse data likelihoods (Papamakarios et al., 2017; Salimans et al., 2017b) , which is surprising at first, as extra bits of important information are provided to the model.

Explaining Likelihood Behaviour One way to understand this seemingly contradictory relationship is to consider the objective we use to train our models. When we train a generative model with maximum likelihood (either exactly or through a lower bound) we are minimizing the empirical approximation of E x,y∼P [-log(P θ (x, y))] which is equivalent to minimizing KL(P (x, y)||P θ (x, y)). Consider now an image x with a discrete label y, which we are trying to model using P θ (x, y). The negative log-likelihood (NLL) objective is:

EQUATION

If we model P θ (y|x) with a uniform distribution over classes, then the second term has a value of log(C) where C is the number of classes. This value is negligible compared to the first term E x∼P [-log(P θ (x))] and therefore the "penalty" for completely ignoring class information is negligible. So it is not surprising that models with strong generative abilities can have limited discriminative power. What makes matters even worse is that the penalty for confident mis-classification can be unbounded. This may also explain why the conditional ELBO is comparable to the unconditional ELBO (Papamakarios et al., 2017) . Another way this can be seen is by thinking of the likelihood as the best lossless compression. When trying to encode an image, the benefit of the label is at most log(C) bits which is small compared to the whole image. While these few bits are important for users, from a likelihood perspective the difference between the correct p(y|x) and a uniform distribution is negligible. This means that when naively training a class-conditional generative model by minimizing E (x,y)∼P [-log(P θ (x|y))], typically discriminative performance as a classifier is very poor.

3.2. OUTLIER DETECTION:

Another issue arises when models trained with maximum likelihood are used to detect outliers. The main issue is that maximum likelihood, which is equivalent to minimizing KL(P (x, y)||P θ (x, y)), is known to have a "mode-covering" behavior. It has been shown recently in (Nalisnick et al., 2019a) that generative models, trained using maximum likelihood, can be quite poor at detecting out-of-distribution example. In fact it has been shown that these models can give higher likelihood values, on average, to datasets different from the test dataset that corresponds to the training data.

Intuitivily one can still hope that a high accuracy conditional generative model would recognize an input conditioned on the wrong class as an outlier, as it was successfully trained to separate these classes. In section 4.2 we show this is not the case in practice.

While (Nalisnick et al., 2019a) focuses its analysis into dataset variance, we propose this is an inherit issue with the likelihood objective. If it is correct then the way conditional generative models are trained is at odds with their desired behaviour. If this is the case, then useful conditional generative model will require a fundamentally different approach.

4. EXPERIMENTS:

We now present a set of experiments designed to test the robustness of conditional generative models. All experiments were performed with a flow model where the likelihood can be computed in closed form as the probability of the latent space embedding (the prior) and a Jacobian correction term; see Sec A.1 for a detailed explanation. Given that we can compute p(x, y) for each class, we can easily compute p(y|x) and classify accordingly. Besides allowing closed-form likelihood computation, the flexibility in choosing the prior distribution was important to conduct various experiments. In our work we used a version of the GLOW model; details of the models and training is in the supplementary material sec. B. We note that the results are not unique to flow models, and we verified that similar phenomenon can be seen when training with the PixelCNN++ autoregressive model (Salimans et al., 2017a) in sec. E.

4.1. TRAINING CONDITIONAL GENERATIVE MODELS:

Here we investigate the ability to train a conditional generative model with good likelihood and accuracy simultaneously. Usually in flow models the prior distribution in latent space z is Gaussian.

For classification we used aclass-conditional mixture of 10 Gaussians p(z|y) = N (µ y , σ 2 y ) We compare three settings: 1) A class-conditional mixture of 10 Gaussians as the prior (Base). 2) A classconditional mixture of 10 Gaussians trained with an additional classification loss term (Reweighted).

3) Our proposed conditional split prior (Split) described in sec. A.4 As we can see, especially on CIFAR10, pushing up the accuracy to values that are still far from stateof-the-art already results in non-negligible deterioration to the likelihood values. This exemplifies how obtaining strong classification accuracy without harming likelihood estimation is still a challenging problem. We note that while the difference between the split prior and re-weighted version is not huge, the split prior achieves better NLL and better accuracy in both experiments. We experimented with various other methods to improve training with limited success, see sec. C in the supplementary material for furture information. Next we show that even conditional generative models which are strong classifiers do not see images with the corrupted labels as outliers. To understand this phenomenon we first note that if we want the correct class to have a probability of at least 1 -δ then it is enough for the corresponding logit to be larger than all the others by log(C) + log1-δ δ where C is the number of classes. For C = 10 and δ = 1e -5 this is about 6, which is negligible relative to the likelihood of the image, which is in the scale of thousands. This means that even for a strong conditional generative model which confidently predicts the correct label, the pair {x i , y w = y i } (where w is the leading incorrect class) cannot be detected as an outlier according to the joint distribution, as the gap log(p(x i |y i )) -log(p(x i |y w )) is much smaller than the variation in likelihood values. In Fig. 3 we show this by plotting the histograms of the likelihood conditioned both on the correct class and on the most likely wrong class over the test set. In other words, in order for log(p(x i |y w )) to be considered an outlier the prediction needs to be extremely confident, much more than we expect it to be, considering test classification error.

4.3. ADVERSARIAL ATTACKS AS WORST CASE ANALYSIS:

We first evaluate the ability of conditional generative models to detect standard attacks, and then try to detect attacks designed to fool the detector (likelihood function). We evalulate both the gradient based Carlini-Wagner L 2 attack (CW-L 2 ) (Carlini & Wagner, 2017b) and the gradient free boundary attack (Brendel et al., 2018) . Results are shown in table 2 on the left. It is interesting to observe the disparity between the CW-L 2 attack, which is easily detectable, and the boundary attack which is much harder to detect. Next we modify our attacks to try to fool the detector as well. With the CW-L 2 attack we follow the modification suggested in (Carlini & Wagner, 2017a) and add an extra loss term det (x ) = max{0, -log(p(x )) -T } where T is the detection threshold. For the boundary attack we turn the C-way classification into a C + 1-way classification by adding another class which is "non-image" and classify any image above the detection threshold as such. We then use a targeted attack to try to fool the network to classify the image into a specific original class. This simple modification to the boundary attack will typically fail because it cannot initialize. The standard attack starts from a random image and all random images are easily detected as "non-image" and therefore do not have the right target class. To address this we start from a randomly chosen image from the target class, ensuring the original image is detected as a real image from the desired class.

Attacking:

From table 2 (right side) we can see that even after the modification CW-L 2 still struggles to fool the detector. The boundary attack, however, succeeds completely on CIFAR10 and fails completely on MNIST, even when it managed to sometimes fool the detector without directly trying. We hypothesize that this is because the area between two images of separate classes, where the boundary attack needs to pass through, is correctly detected as out of distribution only for MNIST and not CIFAR10. We explore this further below.

4.4. AMBIGUOUS INPUTS AS AVERAGE CASE ANALYSIS:

To understand why the learned networks are easily attacked on CIFAR but not on MNIST with the modified boundary attack, we explore the probability density of interpolations between two real images. This is inspired by the fact that the boundary attack proceeds along the line between the attacked image and the initial image. The minimum we would expect from a decent generative model is to detect the intermediate middle images as "non-image" with low likelihood. If this was the case and each class was a disconnected high likelihood region, the boundary attack would have a difficult time when starting from a different class image.

Given images x 0 and x 1 from separate classes y 0 and y 1 and for α ∈ [0, 1] we generate an intermediate image

x α = α • x 1 + (1 -α)x 0

, and run the model on various α values to see the model prediction along the line. For endpoints we sample real images that are classified correctly and are above the detection threshold used previously. See Fig. 1 for interpolation examples from MNIST and CIFAR.

In figure 4 (a) we see the average results for MNIST for 1487 randomly selected pairs. As expected, the likelihood goes down as α moves away from the real images x 0 and x 1 . We also see the probability of both classes drop rapidly as the network predictions become less confident on the intermediate images. Sampling 100 α values uniformly in the range [0, 1] we can also investigate how many of the On CIFAR images, using 1179 pairs, we get a very different picture (see fig. 4 (b) ). Not only does the intermediate likelihood not drop down, it is even higher on average than on the real images albeit to a small degree. In classification we also see a very smooth transition between classes, unlike the sharp drop in the MNIST experiment. Lastly, 100% of the interpolated images lay above the detection threshold and none are detected as a "non-image" (for reference the detection threshold has 78.6% recall on real CIFAR10 test images). This shows that even with good likelihood and reasonable accuracy, the model still "mashes" the classes together, as one can move from one Gaussian to another without passing through low likelihood regions in-between. It also clarifies why the boundary attack is so successful on CIFAR but fails completely on MNIST. We note that the basic attack on MNIST is allowed to pass through these low density areas which is why it sometimes succeeds. In this section, we show that the difference in performance between CIFAR10 and MNIST can largely be attributed to how the entropy in the datasets is distributed, i.e how much the uncertainty in the data distribution is reduced after conditioning on the class label. For MNIST digits, a large source of uncertainty in pixel-space comes from the class label. Given the class, most pixels can be predicted accurately by simply taking the mean of the training set in each class. This is exactly why a linear classifier performs well on MNIST. Conversely on CIFAR10, after conditioning on the class label there still exists considerable uncertainty. Given the class is "cat," there still exists many complicated sources of uncertainty such as where the cat is and how it is posed. In this dataset, a much larger fraction of the uncertainty is not accounted for after conditioning on the label. This is not a function of the domain or the dimensionality of the dataset, it is a function of the dataset itself.

4.5. CLASS-UNRELATED ENTROPY IS TO BLAME:

To empirically verify this, we have designed a dataset which replicates the challenges of CIFAR10 and places them onto a problem of the same discriminative difficulty as MNIST. To achieve this, we simply replaced the black backgrounds of MNIST images with randomly sampled (downsampled and greyscaled) images from CIFAR10. In this dataset, which we call background-MNIST (BG-MNIST), the classification problem is identically predictable from the same set of pixels Published as a conference paper at ICLR 2020 as in standard MNIST but modeling the data density is much more challenging.

To further control the entropy in a fine-grained manner, we convolve the background with a Gaussian blur filter with various bandwidths to remove varying degrees of high frequency information. With high blur, the task begins to resemble standard MNIST and conditional generative models should perform as they do on MNIST. With low and no blur we expect them to behave as they do on CIFAR10.

Table 3 summarizes the performance of conditional generative models on BG-MNIST. We train models with a "Reweighted" discriminative objective as in Section A. The reweighting allows them to perform well as classifiers but the likelihood of their generative component falls to below CIFAR10 levels. More strikingly, now when we interpolate between datapoints we observe behavior identical to our CIFAR10 models. This can be seen in Figure 6 . Thus, we have created a dataset with the discriminative difficulty of MNIST and the generative difficulty of CIFAR10.

MNIST BG-MNIST-5 BG-MNIST-1 BG-MNIST-0 CIFAR10 

5. RELATED WORK:

Despite state of the art performance in many tasks, deep neural networks have been shown to be fragile where small image transformations, (Azulay & Weiss, 2018) or background object transplant (Rosenfeld et al., 2018) can greatly change predictions. In the more challenging case of adversarial pertubations, deep neural networks are known to be vulnerable to adversarial attacks (Akhtar & Mian, 2018) , and while many attempts have been made to train robust models or detect malicious attacks, significant progress towards truly robust models has been made only on MNIST (Schott et al., 2019; Madry et al., 2017) . Even CIFAR10 remains far from being solved from a standpoint of adversarial robustness.

One common belief is that adversarial attacks succeed by moving the data points off the data manifold, and therefore can possibly be detected by a generative model which should assign them low likelihood values. Although this view has been challenged in (Gilmer et al., 2018) , we now discuss how their setting needs to be extended to fully study robustness guarantees of conditional generative models.

Recent work (Song et al., 2018; Frosst et al., 2018; Li et al., 2018) showed that a generative model can detect and defend adversarial attacks. However, there is a caveat when evaluating detectability of adversarial attacks: the attacker needs to be able to attack the detection algorithm as well. Not doing so has been shown to lead to drastically false robustness claims (Carlini & Wagner, 2017a) . In (Li et al., 2018) the authors report difficulties training a high accuracy conditional generative model on CIFAR10, and resort to evaluation on a 2-class classification problem derived from CIFAR10. While they do show robustness similar to our Carlini-Wagner results, they do not apply the boundary attack which we found to break our models on CIFAR10. This highlights the need to utilize a diverse set of attacks. In (Schott et al., 2019) a generative model was used not just for adversarial detection but also robust classification on MNIST, leading to state-of-the-art robust classification accuracy. The method was only shown to work on MNIST, and is very slow at inference time. However, overall it provides an existence proof that conditional generative models can be very robust in practice. In (Ghosh et al., 2019) the authors also use generative models for detection and classification but only show results with the relatively weak FGSM attack, and on simple datasets. As we see in Fig. 1 and discuss in section 4, generative models trained on MNIST can display very different behavior than similar models trained on more challenging data like CIFAR10. This shows how success on MNIST may often not translate to success on other datasets.

6. CONCLUSION:

In this work we explored limitations, both in theory and practice, of using conditional generative models to detect adversarial attacks. Most practical issues arise due to likelihood, the standard objective and evaluation metric for generative models by which probabilities can be computed. We conclude that likelihood-based density modeling and robust classification may fundamentally be at odds with one another as important aspects of the problem are not captured by this training and evaluation metric. This has wide-reaching implications for applications like out-of-distribution detection, adversarial robustness and generalization as well as semi-supervised learning with these models.

A.4 SPLIT PRIOR One reason MLE is bad at capturing the label is because a small number of dimensions have a small effect on the NLL. Fortunately, we can use this property to our advantage. As the contribution of the conditional class information is negligible for the data log likelihood, we choose to model it in its distinct subspace, as proposed by Jacobsen et al. (2019) . Thus, we partition the hidden dimensions z = (z s , z n ) and only try to enforce the low-dimensional z s to be the logits. This has two advantages: 1) we do not enforce class-conditional dimensions to be factorial; and 2) we can explicitly up-weight the loss on this subspace and treat it as standard logits of a discriminative model. A similar approach is also used by semi-supervised VAEs Kingma et al. (2014) . This lets us jointly optimize the data log-likelihood alongside a classification objective without requiring most of the dimensions to be discriminative. 

B IMPLEMENTATION DETAILS:

We pad MNIST with zeros so both datasets are 32x32 and subtract 0.5 from both datasets to have a [-0.5,0.5] range. For data augmentation we do pytorch's random crop with a padding of 4 and 'edge' padding mode, and random horizontal flip for CIFAR10 only.

The model is based on GLOW with 4 levels, affine coupling layers, 1x1 convolution permutations and actnorm in a multi-scale architecture. We choose 128 channels and 12 blocks per level for MNIST and 256 channels and 16 blocks for CIFAR10. In both MNIST and CIFAR10 experiments we double the number of channels with uniforrm(0,1) noise which we scale down to the range [0, 2/256] (taking it into account in the Jacobian term). One major difference is that we do the squeeze operation at the end of each level instead of the beginning, which is what allows us to use 4 levels. This is possible because with the added channels the number of channels is even and the standard splitting is possible before the squeeze operation.

The models are optimized using Adam, for 150 epochs. The initial learning rate is 1e -3, decayed by a factor of 10 every 60 epochs. For the reweighted optimization the objective is

EQUATION

where D is the data dimension (3x32x32 for CIFAR, 1x32x32 for MNIST).

For adversarial detection we use a threshold of 1.4 for MNIST (100% of test data are below the threshold) and 4. for CIFAR10 (78.6% of test images are below the threshold).

C NEGATIVE RESULTS:

In this work explored many ideas in order to achieve better tradeoff between accuracy with little or no impact.

C.1 ROBUST PRIORS:

Since the Gaussian prior is very sensitive to outliers, one idea was that confident miss-classifications carry a strong penalty which might result in "messing" all the classes together. A solution would be to replace the Gaussian with a more robust prior, e.g. Laplace or Cauchy. Another idea we explored is a mixture of Gaussian and Laplace or Cauchy using the same location parameter. In our experiments we did not see any significant difference from the Gaussian prior.

C.2 LABEL SMOOTHING:

Another approach to try to address the same issue is a version of label smoothing. In this new model the Gaussian clusters are a latent variable that is equal to the real label with probability 1 -and uniform on the other labels with probability . Using this will bound the error for confident miss-classification as long as the data is close to one of the Gaussian centers.

C.3 FLOW-GAN:

As we claimed the main issue is with the MLE objective, it seems like a better objective is to optimize KL (p(x, y)||p θ (x, y)) or the Jensen-Shannon divergence as this KL term is highly penalized for miss-classification. It is also more natural when considering robustness against adversarial attacks. Optimizing this directly is hard, but generative adversarial networks (GANs) Goodfellow et al. (2014) in theory should also optimize this objective. Simply training a GAN would not work as we are interested in the likelihood value for adversarial detection and GANs only let you sample and does not give you any information regarding an input image.

Since flow algorithms are bijective, we could combine the two objective as was done in the flow-GAN paper Grover et al. (2018) . We trained this approach with various conditional-GAN alternatives and found it very hard to train. GANs are know to be unstable to train, and combining them with the unstable flow generator is problematic.

D ANALYTICAL COUNTER EXAMPLE::

Assume p(y = 1) = p(y = 0) = q(y = 1) = q(y = 0) = 1/2 and p 

EQUATION

1 C d log λ1 C d 1 C d (1+∆) d (16) R d (1,1+∆) (1 -λ 1 ) C d ((1 + ∆) d -1) log   (1-λ1) C d ((1+∆) d -1) 1 C d (1+∆) d   ≤ λ 1 d log(1 + ∆) < (17)

Proposition 1. For all ( , δ, ∆) there is a distribution p and an approximation q in dimension d = Õ log( δ 1+δ )+log( 1) log(1+∆)

such that KL(q(x, y)||p(x, y)) < , KL(p(x, y)||q(x, y)) < (18)

but with probability greater then 1/3 over samples x ∼ p there is an adversarial example x satisfying:

1. y q (x) = y p (x) with p(y p (x)|x) and q(y q (x)|x) greater or equal to 1 -δ. The original point is classifier correctly and confidently.

2. y q (x) = y(x), y q (x) = y(x). We change the prediction without changing the ground-truth label.

3. q(y q (x)|x) < δ, p(y p (x)|x) > 1 -δ. The classifier is confident in its wrong prediction.

4. ||x -x|| < ∆. We make a small change to the inputs.

5. The density q(x) is greater or equal to the median density, making the attack undetectable by observing q(x).

6. For ∆ < 1 the probability in any radius ball can be made as small as desired.

7. The total variation of the distribution can be made as small as desired.

The last two conditions exclude degenerate trivial counter-exmaples, one where the whole distribution support is in a ∆ radius ball and ∆ does indeed represent a small pertubation. The other condition excludes "pathological" distributions ,e.g. misclassification on a dense zero measure set like the rationals.

Proof. In order to satisfy conditions 1-5, using previous lemmas, it is enough that

1. λ1 λ1+λ2 ≥ 1 -δ 2. 1 1+λ2(1+∆) d ≤ δ 3. λ 1 ≤ 1 -e - 4. λ 1 > 1 (1+∆) d 5. λ 1 < d log(1+∆)

By setting λ 2 = δ 1-δ λ 1 we can easily satisfy condition 1. It is not hard to see that condition 2 is equivalent to λ 1 ≥ 1-δ δ 2 1 (1+∆) d which superseeds condition 4 when δ < 1/2. Condition 3 can be

:

As we model zero padded datasets as described in section A.4, these numbers are not exactly comparable with the literature.

Figures and Tables:

Figure 2: Counter example construction. Shown on the left are the two class data densities, on the right the Bayes-optimal classifier for this problem (assuming λ 1 > λ 2 ) and the model we consider. Despite being almost optimal, the model can be fooled with undetectable adversarial examples (red arrows). Detailed description in section 2.

Figure 3: NLL for images conditioned on the correct class vs the highest probability wrong class.

Figure 4: Average Log likelihoods and class probabilities for interpolations between data points from different classes, x-axis is interpolation coefficient α. The MNIST model behaves as desired and robustly detects interpolated images. The CIFAR10 model, however, fails strikingly and interpolatd images are consistently more likely than true data under the model.

Figure 5: Top: Samples from the BG-MNIST-0 dataset. Bottom: Samples from conditional generative model trained on the dataset. Note how the model has learnd to capture digit identity.

Figure 6: Average log-likelihoods and class probabilities for interpolations between BG-MNIST-0 datapoints. While classification is on par with MNIST models, the likelihood exhibits the same failures as CIFAR10 models.

) where U (a, b) is the uniform distribution on the annulus R d (a, b) = {x ∈ R d : a ≤ ||x|| ≤ b} in dimension d. Lemma 1. For ||x|| < 1 we have The U (a, b) density (when it isn't zero) is 1 C d (b d -a d ) where c d is the volume of the ddimensional unit ball. The proof follows by a simple use of Bayes rule. so by having λ 1 >> λ 2 >> 1 (1+∆) d we can have the model switch wrongfully predictions from y = 0 to y = 1 when we move x from the annulus R d(1, 1 + ∆) to R d (0, 1) Lemma 2. If λ 1 > 1 (1+∆) d and λ 1 < 1 -e -then KL(q(x, y)||P (x, y)) ≤Proof. Using the chain rule for KL divergence, KL(P (x, y)||Q(x, y)) = KL(P (y)||Q(y)) + E y [KL(P (x|y)||Q(x|y))] we get that KL(q(x, y)||P (x, y)) = KL(q(x|y = 0)||P (x|y = 0)).R d (1,1+∆) 1 C d (1 + ∆) d log 1 C d (1+∆) d 1-λ1 C d ((1+∆) d -1) = -log(λ 1 (1 + ∆) d ) (1 + ∆) d(14)+ (1 + ∆) d -1 (1 + ∆) d log (1 + ∆) d -1 (1 -λ 1 )(1 + ∆) d ≤ log 1 1 -λ 1 < (15) Lemma 3. If 1 > λ 1 > 1 (1+∆) d and λ 1 < d log(1+∆) then KL(P (x, y)||q(x, y)) ≤ Proof. Again using the KL chain rule we have KL(P (x|y = 0)||q(x|y = 0)) = λ 1 R d (0,1)

in the supplementary material. Results can be found in table 1. Comparison between different models.

Table content: <table><tr><td colspan="6">MNIST Base Reweight Split CIFAR10 Base Reweight Split</td></tr><tr><td>% Acc</td><td>96.9 99.0</td><td>99.3</td><td>% Acc</td><td>56.8 83.2</td><td>84.0</td></tr><tr><td colspan="2">bits/dim 1 0.95 1.10</td><td>1.00</td><td>bits/dim</td><td>3.47 3.54</td><td>3.53</td></tr></table>

Comparison of attack detection. Percentage of successful and undetected attacks within L 2 -distance of = 1.5 for MNIST and = 33/255 for CIFAR10 for proposed models. Number in parentheses is percentage of attacks that successfully fool the classifier, both detected and undetected.

Table content: <table><tr><td/><td>Classification</td><td/><td>Classification and Detection</td><td/></tr><tr><td>MNIST</td><td>Reweight</td><td>Split</td><td>Reweight</td><td>Split</td></tr><tr><td>CW -L 2</td><td>0% (100%)</td><td>1% (100%)</td><td>17% (100%)</td><td>14% (100%)</td></tr><tr><td colspan="2">Boundary attack 43% (82%)</td><td>36% (80%)</td><td>0% (0%)</td><td>0% (0%)</td></tr><tr><td>CIFAR10</td><td/><td/><td/><td/></tr><tr><td>CW -L 2</td><td>0% (97%)</td><td>0% (0%)</td><td>6% (99%)</td><td>3% (100%)</td></tr><tr><td colspan="2">Boundary attack 67% (100%)</td><td colspan="2">72% (100%) 100% (100%)</td><td>100% (100%)</td></tr></table>

Conditional generative models trained on BG-MNIST. BG-MNIST-X indicates the bandwith of blur applied to CIFAR10 backgrounds.

Table content: <table><tr><td>% Acc</td><td>99</td><td>99</td><td>99</td><td>98</td><td>84</td></tr><tr><td>bits/dim</td><td>1.10</td><td>1.67</td><td>3.30</td><td>4.58</td><td>3.53</td></tr></table>

Using the factorization p(z s , z n |y) = p(z s |y) • p(z n |z s , y) we model p(z s |y) as Gaussian with class conditional mean e i = (0, ..., 0, 1, ...0) and covariance matrix scaled by a constant. The distribution p(z n |z s , y) is modeled as a Gaussian where the mean and variances are a function of y and z n .

Table content: <table/>

Peer review (iclr2020):
Update: I thank the authors* for their response, and have read the other reviews. This paper demonstrates some theoretical and practical limitations on the use of likelihood based generative models for detecting adversarial examples. They construct a simple counterexample showing that there are adversarial examples for an arbitrarily accurate model (as measured by KL) that are not detectable by diminished likelihood of the model (as the dimension increases). Extending the work of Gilmer et al, this proves that there can be no general robustness guarantee for conditional generative models (Bayes classifiers). They provide compelling empirical evidence that while conditional normalizing flows trained on MNIST can be effective in detecting and defending adversarial attacks, these models trained on CIFAR10 are not. Surprisingly, it is shown that linear interpolations between images of different classes yield higher likelihoods for the CIFAR10 models and that class has little impact on model likelihoods. This goes some way in explaining why the detection is not effective on CIFAR, but questions still remain. The paper makes fairly modest claims, but does a good job at demonstrating them and shedding some light on the issue. The experiments are thorough and fit into a growing body of evidence that the likelihoods of normalizing flows and other image based likelihood models may not be that informative or well calibrated, where past work has focused on out-of-distribution detection. My only major complaint with the paper is that it is not clear to what extent the theoretical and practical problems are related. As mentioned in the paper, the counterexample construction depends on the geometry of the data rather than the learning model. It could be that for both the MNIST and CIFAR10 datasets, the geometry is such that robustness garauntees are possible, and that the discrepancy in detection and interpolation arises because the normalizing flow has modeled the MNIST distribution much better than the CIFAR10 distribution. In this case we might hope that using conditional likelihood models for adversarial detection can be made effective, but that effort needs to be placed into improving the modeling capability. It*s not obvious how to probe this distinction, but it would be good if this was given some thought in the paper. Also it would be good to see the attack detection numbers on BG-MNIST. Comments: Difficulty in training conditional generative models: I believe in the two papers you cite the models do not use the label as input, but rather there is a separate model for each class? The overfitting is likely why the models had slightly lower conditional likelihood. As an aside, there are a couple of other examples of conditional normalizing flow models on images that use a mixture of Gaussians in the latent space [1], [2]. eq. 4: In the paper it is said that the second term in eq 4 is at most log(C), because the uniform distribution would have this value and that therefore this is negligibly small in comparison to the other term. Why exactly is this the case, couldn*t the data entropy term be smaller in principle even if it*s larger in practice? Or is the argument that the data entropy term scales with the dimensionality, but the label term does not leading to an imbalance? This could use some clarification. A3: What is meant by ‘While the ground-truth likelihoods for the padded and un-padded datapoints are the same due to independence of the uniform noise and unit density of the noise’ in the appendix section A3? Wouldn*t the ground truth negative log likelihoods would increase by the entropy of the uniform noise? Also, then in the bits per dimension calculations is the dimension the number of unpadded dimensions or the padded dimensions? [1] Izmailov, Pavel, et al. *Semi-Supervised Learning with Normalizing Flows.* Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019. [2] Atanov, Andrei, et al. *Semi-Conditional Normalizing Flows for Semi-Supervised Learning.* Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019.