Less is More: Dimension Reduction Finds On-Manifold Adversarial Examples in Hard-Label Attacks

1 INTRODUCTION . Adversarial examples against deep learning models were originally investigated as blind spots in classification ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . Formal methods for discovering these blind spots emerged , which we denote as gradient-level attacks , and became the first techniques to reach widespread attention within the deep learning community ( Papernot et al. , 2016 ; MoosaviDezfooli et al. , 2015 ; Carlini & Wagner , 2016 ; 2017 ; Chen et al. , 2018 ) . In order to compute the necessary gradient information , such techniques required access to the model parameters and a sizeable query budget . These shortcomings were addressed by the creation of score-level attacks , which only require the confidence values output by the deep learning models ( Fredrikson et al. , 2015 ; Tramèr et al. , 2016 ; Chen et al. , 2017 ; Ilyas et al. , 2018 ) . However , these attacks still rely on models to divulge information that would be impractical to receive in real-world systems . By contrast , hard-label attacks make no assumptions about receiving side information , and only the predicted class is observable , thus providing the weakest , yet most realistic adversarial threat model . These methods , which originated from a random-walk on the decision boundary ( Brendel et al. , 2017 ) , have been carefully refined to offer convergence guarantees ( Cheng et al. , 2019 ) , query efficiency ( Chen et al. , 2019 ; Cheng et al. , 2020 ) , and capability in the physical world Feng et al . ( 2020 ) . Despite the steady improvements of hard-label attacks , open questions persist about their behavior , and adversarial machine learning ( AML ) attacks at large . Adversarial examples were originally assumed to lie in rare pockets of the input space ( Goodfellow et al. , 2014 ) , but this conventional wisdom was later challenged by the boundary tilting assumption ( Tanay & Griffin , 2016 ; Gilmer et al. , 2018 ) , which adopts a “ data-geometric ” view of the input space living on a lower-dimensional manifold . This is supported by Stutz et al . ( 2019 ) , who suggest that regular adversarial examples leave the data manifold , while on-manifold adversarial examples are generalization errors . From a data-geometric perspective , an adversarial example ’ s distance to the manifold primarily describes the amount of semantic features preserved during the attack process . This makes it advantageous to produce on-manifold adversarial examples , since the adversary can exploit the inherent generalization error of the model while producing samples that are semantically similar for humans . However , the true data manifold is either difficult or impossible to describe , and relying solely on approximations of the manifold can lead to the creation of crude adversarial examples ( Stutz et al. , 2019 ) . In this paper , we adopt the boundary-tilting assumption and demonstrate an unexpected benefit of query-efficient zeroth-order attacks , i.e. , attacks enabled by the use of dimensionality reduction techniques . These attacks are more likely to discover on-manifold examples , which we theoretically demonstrate is the result of manifold-gradient mutual information . Our results suggest that this quantity can increase as a function of the data dimensionality . This information leakage leads to adversarial examples that are on-manifold generalization errors . With this knowledge , we empirically demonstrate how to improve hard-label attacks in a generic yet principled way , and potentially re-think their interaction with model robustness and public-facing systems in the near future . For clarity , we provide a block diagram of our claims and experiments in the Appendix ( Section A.3 ) . Our specific contributions are as follows : • Introduction of manifold distance oracle . To create on-manifold examples , the adversary must ( implicitly ) leverage manifold information during the attack phase . We thus propose an informationtheoretic formulation of the noisy manifold distance ( NMD ) oracle , which can explain how zerothorder attacks craft on-manifold examples . We theoretically demonstrate on a Gaussian data model that manifold-gradient mutual information can increase as a function of data dimensionality . We empirically show this is true even on large-scale image datasets such as CIFAR-10 and ImageNet . This finding relates to known behavior in the gradient-level setting , where semantic manifold priors ( e.g. , shapes and textures ) can be leaked from robust models ( Engstrom et al. , 2019 ) . • Reveal new insights of manifold feedback during query-efficient zeroth-order search . In practice , the data manifold is difficult to characterize . We propose the use of three proxies for manifold distance , which all show consistent results in terms of an adversary ’ s ability to search near the manifold . This methodology allows us to empirically demonstrate the connection between dimension reduction , model robustness , and manifold feedback from the model , beyond the known convergence rates tied to dimensionality ( Nesterov & Spokoiny , 2017 ) . Our findings inform how to search closer to the manifold ( Table 1 ) , reduce gradient deviation ( Table 2 ) , and improve query efficiency ( Figure 2 ) in a simple and generic way for hard-label attacks . • Attack-agnostic method for super-pixel grouping . We show that spatial dimension reduction of a decision-based gradient estimate acts as an attack- and knowledge-agnostic method for searching over super-pixels of an image . More importantly , this helps an attacker exploit a model ’ s reaction to salient input changes , leading to samples closer to the manifold compared to the attack on full dimension . As a result , we demonstrate up to 200 % and 340 % success rate improvement for state-of-the-art hard-label attacks HSJA ( Chen et al. , 2019 ) and Sign-OPT attack ( Cheng et al. , 2020 ) , respectively . 2 RELATED WORK . Since the original discovery of adversarial samples against deep models ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) , the prevailing question was why such examples existed . The original assumption was that adversarial examples lived in low-probability pockets of the input space , and were never encountered during parameter optimization ( Szegedy et al. , 2013 ) . This effect was believed to be amplified by the linearity of weight activations in the presence of small perturbations ( Goodfellow et al. , 2014 ) . These assumptions were later challenged by the boundary tilting assumption , which in summary 1 ) asserts that the train and test sets of a model only occupy a sub-manifold of the true data , while the decision boundary lies close to samples on and beyond the sub-manifold ( Tanay & Griffin , 2016 ) , and 2 ) supports the “ data geometric “ view , where high-dimensional geometry of the true data manifold enables a low-probability error set to exist ( Gilmer et al. , 2018 ) . Likewise the boundary tilting assumption describes adversarial samples as leaving the manifold , which has inspired defenses based on projecting such samples back to the data manifold ( Jalal et al. , 2019 ; Samangouei et al. , 2018 ) . However , these approaches were later defeated by adaptive attacks ( Carlini et al. , 2019 ; Carlini & Wagner , 2017 ; Tramer et al. , 2020 ) . We investigate the scenario where an adversary uses zeroth-order information ( i.e. , top-1 label feedback ) to estimate the desired gradient direction ( Cheng et al. , 2020 ; Chen et al. , 2019 ) . Contemporary attacks in this setting are variants of random gradient-free method ( RGF ) ( Nesterov & Spokoiny , 2017 ) , and rely on formulations which convert the top-1 ( hard ) label , which is a step function , into a continuous real-valued function g : Rd → R , which takes search direction θ ∈ Rd and outputs the distance to the nearest adversarial example ( Cheng et al. , 2018 ) . The gradient estimate is conceived as a function of the gradient ∇g and can be estimated with either two samples of information ( SignOPT ) ( Cheng et al. , 2020 ) , or a single point ( HopSkipJumpAttack ) ( Chen et al. , 2019 ) . Details of specific formulations for each attack are provided in Section A.2 of the Appendix . Query efficiency is a persistent desire in the study of hard-label attacks . One clue for achieving efficiency comes from the theory of gradient estimation error and convergence , which shows that the estimation cost is polynomial in d , the dimension of the optimized variable , thus motivating the use of standard dimension-reduction techniques ( Tu et al. , 2019 ) . However , to date it is not completely understood how this relates to traversal through the data manifold . We leverage previous results of the gradient-level setting ( Stutz et al. , 2019 ; Engstrom et al. , 2019 ) to formulate an explanation of manifold leakage during hard-label adversarial attacks . 3 NOISY MANIFOLD DISTANCE ORACLE . Santurkar et al . ( 2019 ) demonstrate that the gradients of robust models have higher visual semantic alignment with the data compared to gradients of standard models . We build on this finding by first assuming that the benign observable data generates from a true lower-dimension distribution . Under the boundary-tilting assumption , this lower-dimension distribution forms a manifold onto which new observations , either benign or adversarial , can be encoded ( Tanay & Griffin , 2016 ) . Likewise , we assume that deep learning models will learn a lower-dimension representation of the observable data , e.g. , feature layers of convolutional neural networks learn to encode training observations onto a low dimension approximate manifold ( Zhang et al. , 2018 ) . When an adversary creates adversarial samples , they are leveraging a pathway that shadows the model gradient , not the true manifold . Thus there is the possibility that adversarial samples are considered “ off-manifold ” , e.g. , can not be expected to generate naturally from the true manifold . However , it is critical for adversarial samples to be as close to the manifold as possible , since on-manifold adversarial examples can exploit the fundamental generalization error of the model ( Stutz et al. , 2019 ) . More formally , we define the notion of manifold distance as follows . Definition 3.1 ( Manifold Distance ) . Consider the benign sample x0 and adversarial counterpart x . Assuming a perfect encoding back to the true manifold φ , the manifold distance is defined as d ( φ ( x0 ) , φ ( x ) ) , where d is a distance function with the domain of the true manifold . Unfortunately , unless the true manifold for a dataset is known , it is impossible to define φ . Instead , a proxy d′ can be used such that d′ ( x , x′ ) ∼ d ( φ ( x ) , φ ( x′ ) ) . In practice , one can implement d′ with any perceptual distance score , such as Learned Perceptual Image Patch Similarity ( Zhang et al. , 2018 ) . If relying on a distance measure d , such as the Lp-norm , an approximate encoder φ′ ( · ) ∼ φ ( · ) can be learned using reconstruction-based training of autoencoders ( Stutz et al. , 2019 ) , or leveraging feature layers of convolutional neural networks ( Zhang et al. , 2018 ) . We are interested in the class of hard-label adversaries that implicitly minimize some proxy of the manifold distance . Given the result of Santurkar et al . ( 2019 ) , the robust model ’ s gradient could be treated as a manifold distance oracle , because it leaks the direction towards its approximate manifold . As a result , the model acts as an oracle responding to queries about manifold distance , or in other words , an implicit proxy for manifold distance , d′ . In the hard-label setting , the data manifold , true gradient , and model parameters are not accessible . Thus we are interested in a decision-based version of the manifold distance oracle , defined as follows . Definition 3.2 ( Noisy Manifold Distance Oracle ) . Consider a manifold distance oracle instantiating d′ , benign sample x0 , and pair of adversarial samples ( x′ , x′′ ) such that d′ ( x0 , x′ ) < d′ ( x0 , x′′ ) , e.g. , x′ is considered on-manifold while x′′ is not . In the hard-label setting , the noisy manifold distance ( NMD ) oracle instantiates d′′ such that d′′ ( x0 , x′ ) = 0 and d′′ ( x0 , x′′ ) = 1 . During a hard-label attack , the adversary searches in a direction that minimizes perceptual distance to the original sample . Concurrently , the adversary can be said to implicitly minimize the expected output of the NMD oracle , which is a binary indicator that a sample is on-manifold or not . Without knowledge of the true ( or approximate ) manifold , this requires careful selection of the search direction from the current sample . Since the search direction of contemporary hard-label attacks is synthesized over expectation of a ball around the adversarial sample , we are interested in search directions such as x0 − x′ which minimize the expected distance to the manifold . To formalize the entailed information in the NMD oracle , we turn to a standard result in data processing , which states the following : Definition 3.3 ( Data Processing Inequality ( DPI ) ( Beaudry & Renner , 2012 ) ) . If three random variables form the Markov chain X → Y → Z , then their mutual information ( MI ) has the relation I ( X ; Y ) > I ( X ; Z ) . We assume the data manifoldM , the input gradient G , and the hard-label gradient estimate G̈ will form the Markov chainM→ G → G̈ . This assumption is reasonable due to the observations by Santurkar et al . ( 2019 ) ; modifying the sampled data manifold ( e.g. , by adding adversarial samples through saddle-point optimization ) causally induces a smoother loss surface , which imposes its own gradient distribution . Likewise , the true gradient and gradient estimate of hard-label attack are causally linked due to the estimate ’ s bounded variance ( Cheng et al. , 2020 ) . If I ( M , G ) is larger for adversarially robust models , by Definition 3.3 the upper bound on I ( M , G̈ ) is larger , which means more manifold information could be leaked in the noisy gradient . This information could be used to search in the direction where d′′ is minimized in expectation , leading towards on-manifold examples . However , DPI only offers an upper bound , thus the distance decrease is not guaranteed , only suggested . In the information theoretic sense , does this mean the gradients of models robust in an -ball around each sample can reveal more information about the distance to training data than standard models ? An immediate follow-up concern is whether other factors can influence the model to reveal this information , such as the problem dimensionality . As a first step we posit the following hypothesis : Hypothesis 1 . Consider the manifold distributionM which can generate data to train a natural model with gradient distribution G , and train robust model with smoothed gradient distribution G′ . We posit that their manifold-gradient mutual information I has the relation I ( M , G′ ) ≥ I ( M , G ) . In order to empirically verify Hypothesis 1 , we must parameterize the notion of model robustness while solving for I ( M , G ) , given an arbitrary gradient distribution G and manifold distributionM . Schmidt et al . ( 2018 ) have shown that robust training requires additional data as a function of the data dimensionality . We leverage the data model and results from Schmidt et al . ( 2018 ) to derive an analytical solution for I ( M , G ) , since we can parameterize model robustness as a function of data size and dimensionality . Consequently , the remainder of our theoretical analysis assumes a Gaussian mixture data model . Definition 3.4 ( Data model and optimal weights ( Schmidt et al. , 2018 ) ) . Let µ ∈ Rd be the per-class centers ( means ) and let σ > 0 be the variance parameter . Then the ( µ , σI ) -Gaussian model is defined by the following distribution over ( x , y ) ∈ Rd × { ±1 } : First , draw a label y ∈ { ±1 } uniformly at random . Then sample the data point x ∈ Rd from N ( y · µ , σI ) . Definition 3.5 ( Optimal classification weight ( Schmidt et al. , 2018 ) ) . Fix σ ≤ c1d 1 4 for the universal constant c1 , and samples ( x1 , y1 ) , · · · , ( xn , yn ) drawn i.i.d from the ( µ , σI ) -Gaussian model with ||µ|| = √ d ( i.e. , µk = 1 for all dimensions k ∈ { 0 , . . . , d } ) . Schmidt et al . ( 2018 ) prove that the weight setting ŵ = 1n ∑n i=1 yixi yields an l ∞-robust classification error of at most 1 % for the linear classifier fŵ : Rd → { ±1 } instantiated as fŵ ( x ) = sign ( ŵTx ) if n ≥ { 1 , for ≤ 1 4 d− 1 4 c2 2 √ d , for 1 4 d− 1 4 ≤ ≤ 1 4 , ( 1 ) for a universal constant c2 . Note that the instantiation of ŵ must change with choice of and d. We can leverage the weight settings as a function of n and d to give a definition of manifold-gradient mutual information .

Summary: This paper concerns zeroth-order hard-label adversarial attacks on machine learning models, and in particular, how to strengthen them through leveraging manifold information. The core idea is inspired from a series of intuitions and observations around the fact that the gradient of a robust model has more information regarding the underlying low-dimensional manifold of the data, compared to that of a naturally trained model. Author(s) have built upon this inspiration and proposed the noisy manifold distance oracle, which can leak manifold information to the adversary during forging its attack. Author(s) have also implemented their method on a number of real-world datasets.