On the Need for Topology-Aware Generative Models for Manifold-Based Defenses

1 INTRODUCTION . Machine-learning ( ML ) algorithms , especially deep-neural networks ( DNNs ) , have had resounding success in several domains . However , adversarial examples have hindered their deployment in safety-critical domains , such as autonomous driving and malware detection . Adversarial examples are constructed by an adversary adding a small perturbation to a data-point so that it is misclassified . Several algorithms for constructing adversarial examples exist in the literature ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ; Goodfellow et al. , 2014b ; Kurakin et al. , 2016a ; Carlini & Wagner , 2017 ; Madry et al. , 2017 ; Papernot et al. , 2017 ) . Numerous defenses for adversarial examples also have been explored ( Kurakin et al. , 2016b ; Guo et al. , 2017 ; Sinha et al. , 2017 ; Song et al. , 2017 ; Tramèr et al. , 2017 ; Xie et al. , 2017 ; Dhillon et al. , 2018 ; Raghunathan et al. , 2018 ; Cohen et al. , 2019 ; Dubey et al. , 2019 ) . In this paper , we focus on “ manifold-based ” defenses ( Ilyas et al. , 2017 ; Samangouei et al. , 2018 ) . The general idea in these defenses is to “ pull back ” the data point into the data manifold before classification . These defenses leverage the fact that , in several domains , natural data lies in a low-dimensional manifold ( henceforth referred to as the manifold assumptions ) ( Zhu & Goldberg , 2009 ) . The data distribution and hence actual manifold that the natural data lies in is usually unknown , so these defenses use a generative model to “ approximate ” the data distribution . Generative models attempt to learn to generate data according to the underlying data distribution . ( The input to a generative model is usually random noise from a known distribution , such as Gaussian or uniform . ) There are various types of generative models in the literature , such as variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) , generative adversarial network ( GAN ) ( Goodfellow et al. , 2014a ) and reversible generative models , e.g. , real-valued non-volume preserving transform ( Real NVP ) ( Dinh et al. , 2016 ) . This paper addresses the following question : Do manifold-based defenses need to be aware of the topology of the underlying data manifold ? In this paper , we suggest the answer to this question is yes . We demonstrate that if the generative model does not capture the topology of the underlying manifold , it can adversely affect these defenses . In these cases , the underlying generative model is being used as an approximation of the underlying manifold . We believe this opens a rich avenue for future work on using topology-aware generative models for defense to adversarial examples . Contributions and Roadmap . We begin with a brief description of related work in Section 2 . Section 3 provides the requisite mathematical background . Our main theoretical results are provided in Section 4 . Informally , our result says that if the generative model is not topology-aware , it can lead to a ” topological mismatch ” between the distribution induced by the generative model and the actual distribution . Section 5 describes our experimental verification of our theoretical results and investigates their ramifications on a manifold-based defenses called Invert-and-Classify ( INC ) ( Ilyas et al. , 2017 ; Samangouei et al. , 2018 ) . 2 RELATED WORK . 2.1 GENERATIVE MODELS . As a method for sampling high-dimensional data , generative models find applications in various fields in applied math and engineering , e.g. , image processing , reinforcement learning , etc . Methods for learning data-generating distribution with neural networks include well-known examples of Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) and variations of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014a ; Radford et al. , 2015 ; Zhao et al. , 2016 ) . These generative models learn how to map latent variables into generated samples . The VAE is a variational Bayesian approach , so it approximates a posterior distribution over latent vectors ( given training samples ) by a simpler variational distribution . Similar to other variational Bayesian methods , VAE tries to minimize the Kullback–Leibler divergence between the posterior distribution and the variational distribution by minimizing the reconstruction error of the autoencoder . GANs represent another approach to learning how to transform latent vectors into samples . Unlike other approaches , the GAN learns the target distribution by training two networks – generator and discriminator – simultaneously . In addition to generating plausible samples , some generative models construct bijective relations between latent vector and generated samples , so that the probability density of the generated sample can be estimated . Due to their bijective nature , such generative models are called to be reversible . Some examples are normalizing flow ( Rezende & Mohamed , 2015 ) , Masked Autoregressive Flow ( MAF ) ( Papamakarios et al. , 2017 ) , Real NVP ( Dinh et al. , 2016 ) , and Glow ( Kingma & Dhariwal , 2018 ) . 2.2 APPLICATIONS OF GENERATIVE MODELS IN ADVERSARIAL MACHINE LEARNING . The DNN-based classifier has been shown to be vulnerable to adversarial attacks ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014b ; Moosavi-Dezfooli et al. , 2016 ; Papernot et al. , 2016 ; Madry et al. , 2017 ) . Several hypothesis try explaining such vulnerability ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014b ; Tanay & Griffin , 2016 ; Feinman et al. , 2017 ) , and one explanation is that the adversarial examples lie far away from the data manifold . This idea leads to defenses making use of the geometry learned from the dataset – by projecting the input to the nearest point in the data manifold . To learn a manifold from a given dataset , generative models can be exploited . The main idea is to approximate the data-generating distribution with a generative model , to facilitate searching over data manifold by searching over the space of latent vectors . The term Invert-and-Classfy ( INC ) was coined to describe this type of defense ( Ilyas et al. , 2017 ) , and different types of generative models were tried to detect adversarial examples ( Ilyas et al. , 2017 ; Song et al. , 2017 ; Samangouei et al. , 2018 ) . Usually , the projection is done by searching the latent vector that minimizes the geometric distance ( Ilyas et al. , 2017 ; Samangouei et al. , 2018 ) . However , despite the promising theoretical background , all of those methods are still vulnerable ( Athalye et al. , 2018 ; Ilyas et al. , 2017 ) . 3 BACKGROUND . We formally describe data generation , based on the well-known manifold assumption ; data lies close to a manifold whose intrinsic dimension is much lower than that of the ambient space . In our model of data generation , we provide a formal definition of data-generating manifold M on which the data-generating distribution lies such that M conforms to the manifold assumption . 3.1 REQUIREMENTS . Real-world data tends to be noisy , so the data does not easily correspond to an underlying manifold . We first focus on an ideal case where data is generated solely from the manifold M without noise . In the setting of classification with l labels , we consider manifolds M1 , . . . , Ml ⊂ Rn that correspond to the generation of data in each class i ∈ { 1 , . . . , l } , respectively . We assume those manifolds are pair-wise disjoint , i.e. , Mi ∩Mj = ∅ for any i /= j.We set the data-generating manifold M as the disjoint union of those manifolds , M = ⊍li=1Mi . We assume M to be a compact Riemannian manifold with a volume measure dM induced by its Riemannian metric . When a density function pM defined on M satisfies some requirements , it is possible to compute probabilities over M via ∫x∈M pM ( x ) dM ( x ) . We call such M equipped with pM an dM as a data-generating manifold . We refer to Appendix A and Appendix D.1 for details about definitions and requirements on pM . In practice , data generation is affected by noise , so not all data lie on the data-generating manifold . Therefore , we incorporate the noise as an artifact of data-generation and extend the density pM on M to the density p on the entire Rn by assigning local noise densities on M . We consider a procedure that ( 1 ) samples a point xo from M first , and ( 2 ) adds a noise vector n to get an observed point x̂ = xo +n . Here , the noise n is a random vector sampled from a probability distribution , centered at xo , whose noise density function is νxo , satisfying νx ( n ) = νx ( x̂ − x ) = pM ( x̂∣xo = x ) . 3.2 EXTENDING DENSITY . When M is equipped with a density function pM and a measure dM that we can integrate over M , we can compute the density after random noise is added as follows . p ( x̂ ) = ∫ x∈M νx ( x̂ − x ) p ( x ) dM ( x ) ( 1 ) Since νx ( x̂ − x ) is a function on x when x̂ is fixed , computing this integration can be viewed as the computing expectation of a real-valued function defined on M . Computing such expectation has been explored in Pennec ( 1999 ) . A demonstrative example is provided in Appendix B , and this extension is further discussed in Appendix D.2 . 3.3 GENERATIVE MODELS . A generative model tries to find a statistical model for joint density p ( x , y ) ( Ng & Jordan , 2002 ) . We mainly discuss a specific type that learns a transform from one distribution DZ to another target distribution DX . Commonly , a latent vector z ∼ DZ is sampled from a simpler distribution , e.g. , Gaussian , then a pre-trained deterministic function G maps to a sample x = G ( z ) . Specifically , we focus on reversible generative models to facilitate the comparison between the density of generated samples and the target density . In this approach , the dimensions of latent vectors are set to be the same as those of the samples to be generated . Also , for a given x , the density of x is estimated by the change of variable formula ( equation ( 2 ) in Section 5.1 ) . 3.4 INVERT AND CLASSIFY ( INC ) APPROACH FOR ROBUST CLASSIFICATION . As the data-generating manifold M contains class-wise disjoint manifolds , there is a classifier f on Rn separating these manifolds . If f separates the manifolds of M , any misclassified point should lie out of M . Therefore , to change a correct classification near a manifold , any adversary would pull a sample further out of the manifold . By projecting misclassified points to the nearest manifold , we may expect the classification to be corrected by the projection . The INC method ( Ilyas et al. , 2017 ; Samangouei et al. , 2018 ) implements this using a generative model . The main idea of INC is to invert the perturbed sample by projecting to the nearest point on the data-generating manifold . Ideally , the data-generating manifold M is accessible . For any point ( x̂ , y ) with f ( x̂ ) /= y , out-of-manifold perturbation is reduced by projecting x̂ to x∗ on M . The manifold M is unknown in practice . However , as M is the data-generating manifold of DX , a generative model G for DX is trained to approximate M . Then , searching over M is replaced by searching over latent vectors of G. More details about INC implementations are described in Section 5.1 .

Summary: This paper argues that defenses against adversarial attacks need to be stronger than they currently are. Defenses that use generative models assume that there exists a manifold of data that is modeled by a trained generative model that can be used to project any out-of-manifold data unto the manifold. However, this work argues that if the generative model does not model the topology of the manifold, it can still be fooled by an adversarial example. They argue that a generative model needs to be at least aware of the number of connected components of the data-generating manifold. If the number of connected components does not match, based on theorem 2, Corollary 1 argues that a generative model can generate an adversarial example that does not exist in the data-generating manifold.