Less is More: Dimension Reduction Finds On-Manifold Adversarial Examples in Hard-Label Attacks

1 INTRODUCTION . Adversarial examples against deep learning models were originally investigated as blind spots in classification ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . Formal methods for discovering these blind spots emerged , which we denote as gradient-level attacks , and became the first techniques to reach widespread attention within the deep learning community ( Papernot et al. , 2016 ; MoosaviDezfooli et al. , 2015 ; Carlini & Wagner , 2016 ; 2017 ; Chen et al. , 2018 ) . In order to compute the necessary gradient information , such techniques required access to the model parameters and a sizeable query budget . These shortcomings were addressed by the creation of score-level attacks , which only require the confidence values output by the deep learning models ( Fredrikson et al. , 2015 ; Tramèr et al. , 2016 ; Chen et al. , 2017 ; Ilyas et al. , 2018 ) . However , these attacks still rely on models to divulge information that would be impractical to receive in real-world systems . By contrast , hard-label attacks make no assumptions about receiving side information , and only the predicted class is observable , thus providing the weakest , yet most realistic adversarial threat model . These methods , which originated from a random-walk on the decision boundary ( Brendel et al. , 2017 ) , have been carefully refined to offer convergence guarantees ( Cheng et al. , 2019 ) , query efficiency ( Chen et al. , 2019 ; Cheng et al. , 2020 ) , and capability in the physical world Feng et al . ( 2020 ) . Despite the steady improvements of hard-label attacks , open questions persist about their behavior , and adversarial machine learning ( AML ) attacks at large . Adversarial examples were originally assumed to lie in rare pockets of the input space ( Goodfellow et al. , 2014 ) , but this conventional wisdom was later challenged by the boundary tilting assumption ( Tanay & Griffin , 2016 ; Gilmer et al. , 2018 ) , which adopts a “ data-geometric ” view of the input space living on a lower-dimensional manifold . This is supported by Stutz et al . ( 2019 ) , who suggest that regular adversarial examples leave the data manifold , while on-manifold adversarial examples are generalization errors . From a data-geometric perspective , an adversarial example ’ s distance to the manifold primarily describes the amount of semantic features preserved during the attack process . This makes it advantageous to produce on-manifold adversarial examples , since the adversary can exploit the inherent generalization error of the model while producing samples that are semantically similar for humans . However , the true data manifold is either difficult or impossible to describe , and relying solely on approximations of the manifold can lead to the creation of crude adversarial examples ( Stutz et al. , 2019 ) . In this paper , we adopt the boundary-tilting assumption and demonstrate an unexpected benefit of query-efficient zeroth-order attacks , i.e. , attacks enabled by the use of dimensionality reduction techniques . These attacks are more likely to discover on-manifold examples , which we theoretically demonstrate is the result of manifold-gradient mutual information . Our results suggest that this quantity can increase as a function of the data dimensionality . This information leakage leads to adversarial examples that are on-manifold generalization errors . With this knowledge , we empirically demonstrate how to improve hard-label attacks in a generic yet principled way , and potentially re-think their interaction with model robustness and public-facing systems in the near future . For clarity , we provide a block diagram of our claims and experiments in the Appendix ( Section A.3 ) . Our specific contributions are as follows : • Introduction of manifold distance oracle . To create on-manifold examples , the adversary must ( implicitly ) leverage manifold information during the attack phase . We thus propose an informationtheoretic formulation of the noisy manifold distance ( NMD ) oracle , which can explain how zerothorder attacks craft on-manifold examples . We theoretically demonstrate on a Gaussian data model that manifold-gradient mutual information can increase as a function of data dimensionality . We empirically show this is true even on large-scale image datasets such as CIFAR-10 and ImageNet . This finding relates to known behavior in the gradient-level setting , where semantic manifold priors ( e.g. , shapes and textures ) can be leaked from robust models ( Engstrom et al. , 2019 ) . • Reveal new insights of manifold feedback during query-efficient zeroth-order search . In practice , the data manifold is difficult to characterize . We propose the use of three proxies for manifold distance , which all show consistent results in terms of an adversary ’ s ability to search near the manifold . This methodology allows us to empirically demonstrate the connection between dimension reduction , model robustness , and manifold feedback from the model , beyond the known convergence rates tied to dimensionality ( Nesterov & Spokoiny , 2017 ) . Our findings inform how to search closer to the manifold ( Table 1 ) , reduce gradient deviation ( Table 2 ) , and improve query efficiency ( Figure 2 ) in a simple and generic way for hard-label attacks . • Attack-agnostic method for super-pixel grouping . We show that spatial dimension reduction of a decision-based gradient estimate acts as an attack- and knowledge-agnostic method for searching over super-pixels of an image . More importantly , this helps an attacker exploit a model ’ s reaction to salient input changes , leading to samples closer to the manifold compared to the attack on full dimension . As a result , we demonstrate up to 200 % and 340 % success rate improvement for state-of-the-art hard-label attacks HSJA ( Chen et al. , 2019 ) and Sign-OPT attack ( Cheng et al. , 2020 ) , respectively . 2 RELATED WORK . Since the original discovery of adversarial samples against deep models ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) , the prevailing question was why such examples existed . The original assumption was that adversarial examples lived in low-probability pockets of the input space , and were never encountered during parameter optimization ( Szegedy et al. , 2013 ) . This effect was believed to be amplified by the linearity of weight activations in the presence of small perturbations ( Goodfellow et al. , 2014 ) . These assumptions were later challenged by the boundary tilting assumption , which in summary 1 ) asserts that the train and test sets of a model only occupy a sub-manifold of the true data , while the decision boundary lies close to samples on and beyond the sub-manifold ( Tanay & Griffin , 2016 ) , and 2 ) supports the “ data geometric “ view , where high-dimensional geometry of the true data manifold enables a low-probability error set to exist ( Gilmer et al. , 2018 ) . Likewise the boundary tilting assumption describes adversarial samples as leaving the manifold , which has inspired defenses based on projecting such samples back to the data manifold ( Jalal et al. , 2019 ; Samangouei et al. , 2018 ) . However , these approaches were later defeated by adaptive attacks ( Carlini et al. , 2019 ; Carlini & Wagner , 2017 ; Tramer et al. , 2020 ) . We investigate the scenario where an adversary uses zeroth-order information ( i.e. , top-1 label feedback ) to estimate the desired gradient direction ( Cheng et al. , 2020 ; Chen et al. , 2019 ) . Contemporary attacks in this setting are variants of random gradient-free method ( RGF ) ( Nesterov & Spokoiny , 2017 ) , and rely on formulations which convert the top-1 ( hard ) label , which is a step function , into a continuous real-valued function g : Rd → R , which takes search direction θ ∈ Rd and outputs the distance to the nearest adversarial example ( Cheng et al. , 2018 ) . The gradient estimate is conceived as a function of the gradient ∇g and can be estimated with either two samples of information ( SignOPT ) ( Cheng et al. , 2020 ) , or a single point ( HopSkipJumpAttack ) ( Chen et al. , 2019 ) . Details of specific formulations for each attack are provided in Section A.2 of the Appendix . Query efficiency is a persistent desire in the study of hard-label attacks . One clue for achieving efficiency comes from the theory of gradient estimation error and convergence , which shows that the estimation cost is polynomial in d , the dimension of the optimized variable , thus motivating the use of standard dimension-reduction techniques ( Tu et al. , 2019 ) . However , to date it is not completely understood how this relates to traversal through the data manifold . We leverage previous results of the gradient-level setting ( Stutz et al. , 2019 ; Engstrom et al. , 2019 ) to formulate an explanation of manifold leakage during hard-label adversarial attacks . 3 NOISY MANIFOLD DISTANCE ORACLE . Santurkar et al . ( 2019 ) demonstrate that the gradients of robust models have higher visual semantic alignment with the data compared to gradients of standard models . We build on this finding by first assuming that the benign observable data generates from a true lower-dimension distribution . Under the boundary-tilting assumption , this lower-dimension distribution forms a manifold onto which new observations , either benign or adversarial , can be encoded ( Tanay & Griffin , 2016 ) . Likewise , we assume that deep learning models will learn a lower-dimension representation of the observable data , e.g. , feature layers of convolutional neural networks learn to encode training observations onto a low dimension approximate manifold ( Zhang et al. , 2018 ) . When an adversary creates adversarial samples , they are leveraging a pathway that shadows the model gradient , not the true manifold . Thus there is the possibility that adversarial samples are considered “ off-manifold ” , e.g. , can not be expected to generate naturally from the true manifold . However , it is critical for adversarial samples to be as close to the manifold as possible , since on-manifold adversarial examples can exploit the fundamental generalization error of the model ( Stutz et al. , 2019 ) . More formally , we define the notion of manifold distance as follows . Definition 3.1 ( Manifold Distance ) . Consider the benign sample x0 and adversarial counterpart x . Assuming a perfect encoding back to the true manifold φ , the manifold distance is defined as d ( φ ( x0 ) , φ ( x ) ) , where d is a distance function with the domain of the true manifold . Unfortunately , unless the true manifold for a dataset is known , it is impossible to define φ . Instead , a proxy d′ can be used such that d′ ( x , x′ ) ∼ d ( φ ( x ) , φ ( x′ ) ) . In practice , one can implement d′ with any perceptual distance score , such as Learned Perceptual Image Patch Similarity ( Zhang et al. , 2018 ) . If relying on a distance measure d , such as the Lp-norm , an approximate encoder φ′ ( · ) ∼ φ ( · ) can be learned using reconstruction-based training of autoencoders ( Stutz et al. , 2019 ) , or leveraging feature layers of convolutional neural networks ( Zhang et al. , 2018 ) . We are interested in the class of hard-label adversaries that implicitly minimize some proxy of the manifold distance . Given the result of Santurkar et al . ( 2019 ) , the robust model ’ s gradient could be treated as a manifold distance oracle , because it leaks the direction towards its approximate manifold . As a result , the model acts as an oracle responding to queries about manifold distance , or in other words , an implicit proxy for manifold distance , d′ . In the hard-label setting , the data manifold , true gradient , and model parameters are not accessible . Thus we are interested in a decision-based version of the manifold distance oracle , defined as follows . Definition 3.2 ( Noisy Manifold Distance Oracle ) . Consider a manifold distance oracle instantiating d′ , benign sample x0 , and pair of adversarial samples ( x′ , x′′ ) such that d′ ( x0 , x′ ) < d′ ( x0 , x′′ ) , e.g. , x′ is considered on-manifold while x′′ is not . In the hard-label setting , the noisy manifold distance ( NMD ) oracle instantiates d′′ such that d′′ ( x0 , x′ ) = 0 and d′′ ( x0 , x′′ ) = 1 . During a hard-label attack , the adversary searches in a direction that minimizes perceptual distance to the original sample . Concurrently , the adversary can be said to implicitly minimize the expected output of the NMD oracle , which is a binary indicator that a sample is on-manifold or not . Without knowledge of the true ( or approximate ) manifold , this requires careful selection of the search direction from the current sample . Since the search direction of contemporary hard-label attacks is synthesized over expectation of a ball around the adversarial sample , we are interested in search directions such as x0 − x′ which minimize the expected distance to the manifold . To formalize the entailed information in the NMD oracle , we turn to a standard result in data processing , which states the following : Definition 3.3 ( Data Processing Inequality ( DPI ) ( Beaudry & Renner , 2012 ) ) . If three random variables form the Markov chain X → Y → Z , then their mutual information ( MI ) has the relation I ( X ; Y ) > I ( X ; Z ) . We assume the data manifoldM , the input gradient G , and the hard-label gradient estimate G̈ will form the Markov chainM→ G → G̈ . This assumption is reasonable due to the observations by Santurkar et al . ( 2019 ) ; modifying the sampled data manifold ( e.g. , by adding adversarial samples through saddle-point optimization ) causally induces a smoother loss surface , which imposes its own gradient distribution . Likewise , the true gradient and gradient estimate of hard-label attack are causally linked due to the estimate ’ s bounded variance ( Cheng et al. , 2020 ) . If I ( M , G ) is larger for adversarially robust models , by Definition 3.3 the upper bound on I ( M , G̈ ) is larger , which means more manifold information could be leaked in the noisy gradient . This information could be used to search in the direction where d′′ is minimized in expectation , leading towards on-manifold examples . However , DPI only offers an upper bound , thus the distance decrease is not guaranteed , only suggested . In the information theoretic sense , does this mean the gradients of models robust in an -ball around each sample can reveal more information about the distance to training data than standard models ? An immediate follow-up concern is whether other factors can influence the model to reveal this information , such as the problem dimensionality . As a first step we posit the following hypothesis : Hypothesis 1 . Consider the manifold distributionM which can generate data to train a natural model with gradient distribution G , and train robust model with smoothed gradient distribution G′ . We posit that their manifold-gradient mutual information I has the relation I ( M , G′ ) ≥ I ( M , G ) . In order to empirically verify Hypothesis 1 , we must parameterize the notion of model robustness while solving for I ( M , G ) , given an arbitrary gradient distribution G and manifold distributionM . Schmidt et al . ( 2018 ) have shown that robust training requires additional data as a function of the data dimensionality . We leverage the data model and results from Schmidt et al . ( 2018 ) to derive an analytical solution for I ( M , G ) , since we can parameterize model robustness as a function of data size and dimensionality . Consequently , the remainder of our theoretical analysis assumes a Gaussian mixture data model . Definition 3.4 ( Data model and optimal weights ( Schmidt et al. , 2018 ) ) . Let µ ∈ Rd be the per-class centers ( means ) and let σ > 0 be the variance parameter . Then the ( µ , σI ) -Gaussian model is defined by the following distribution over ( x , y ) ∈ Rd × { ±1 } : First , draw a label y ∈ { ±1 } uniformly at random . Then sample the data point x ∈ Rd from N ( y · µ , σI ) . Definition 3.5 ( Optimal classification weight ( Schmidt et al. , 2018 ) ) . Fix σ ≤ c1d 1 4 for the universal constant c1 , and samples ( x1 , y1 ) , · · · , ( xn , yn ) drawn i.i.d from the ( µ , σI ) -Gaussian model with ||µ|| = √ d ( i.e. , µk = 1 for all dimensions k ∈ { 0 , . . . , d } ) . Schmidt et al . ( 2018 ) prove that the weight setting ŵ = 1n ∑n i=1 yixi yields an l ∞-robust classification error of at most 1 % for the linear classifier fŵ : Rd → { ±1 } instantiated as fŵ ( x ) = sign ( ŵTx ) if n ≥ { 1 , for ≤ 1 4 d− 1 4 c2 2 √ d , for 1 4 d− 1 4 ≤ ≤ 1 4 , ( 1 ) for a universal constant c2 . Note that the instantiation of ŵ must change with choice of and d. We can leverage the weight settings as a function of n and d to give a definition of manifold-gradient mutual information .

Summary: The paper addresses hard-label adversarial attacks from a geometric/manifold perspective. Specifically, it examines examples that live close to or on the data manifold. The paper presents a noisy manifold distance based on information-theoretic considerations and proposes three ways to approximate it (taking into account that the data manifold is unknown). Experimental results of HSJA and Sign-OPT attacks on CIFAR-10 and ImageNet are presented.