NETWORK ROBUSTNESS TO PCA PERTURBATIONS

1 INTRODUCTION . The reliability of deep neural networks ( DNNs ) has been undermined by adversarial examples : small perturbations to inputs that deceive the network ( e.g. , Goodfellow et al . ( 2015 ) ) . A key step in recovering DNN reliability is identifying input features for which the network is robust . Existing work focuses on the input values , the lowest-level features , to evaluate the network robustness . For example , a lot of work analyzes networks ’ robustness to neighborhoods consisting of all inputs at a certain distance from a given input ( e.g. , Boopathy et al . ( 2019 ) ; Katz et al . ( 2017 ) ; Salman et al . ( 2019 ) ; Singh et al . ( 2019a ) ; Tjeng et al . ( 2019 ) ; Wang et al . ( 2018 ) ) . Despite the variety of approaches introduced to analyze robustness , the diameter ( controlling the neighborhood size ) of the provably robust neighborhoods is often very small . This may suggest an inherent barrier of the robustness of DNNs to distance-based neighborhoods . To illustrate , consider Figure 1 ( a ) and Figure 1 ( b ) – which are visibly the same but in fact each of their pixels differs by = 0.026 . That is the maximal one for which the L∞ ball B ( x ) ( x is Figure 1 ( a ) ) was proven robust by ERAN ( Singh et al. , 2018 ; Gehr et al. , 2018 ) , a state-of-the-art robustness analyzer . Feature-defined neighborhoods We propose to analyze network robustness to perturbations of high-level input features . A small perturbation to a feature translates to changes of multiple input entries ( e.g. , image pixels ) and as such may produce visible perturbations . To illustrate , consider a neighborhood around Figure 1 ( a ) in which only the background pixels can change their color . It turns out that , for this neighborhood , ERAN – the same robustness analyzer – is able to prove a neighborhood which has 10672x more images . Figure 1 ( c ) shows a maximally perturbed image in this neighborhood , and Figure 1 ( d ) illustrates two other images in it . These images are visibly different from Figure 1 ( a ) . Proving such neighborhood robust , for many inputs , can suggest that the network is robust to background color perturbations , thereby provide insights to the patterns the network learned . Key idea : robust features An inherent challenge in finding robust feature-defined neighborhoods is automatically finding good candidate features ( e.g. , background color ) . Part of this challenge stems from the substantial running time of any robustness analyzer on a single neighborhood . This makes brute-force search of feature-defined neighborhoods for a large number of features and inputs futile . We propose a sampling approach to identify features which are likely to be robust for many inputs . We call these robust features . We experimentally observe that our robust features generalize to unseen inputs , even though they were determined from a ( small ) set of inputs . δ ( x ) . ( e ) A traffic light image from Imagenet , correctly classified by Inception-V3 . ( f ) Our feature-based adversarial example classified by the same model as crane . ( g ) A visualization of the difference . ( h ) Perturbations to the image x using the second PCA feature ( whose semantic meaning is sky color ) . Feature-guided adversarial examples Dually to robust features , we define and identify weak features . We show how to exploit them in a simple yet effective black-box attack . Our attack perturbs a given input based on the weak features and then greedily reduces the number of modified pixels to obtain an adversarial example which minimizes the L2 distance from the original input . Figure 1 illustrates our attack on ImageNet with the Inception-V3 architecture ( Szegedy et al. , 2016 ) . Figure 1 ( e ) shows a traffic light image ( correctly classified ) , Figure 1 ( f ) shows our feature-guided adversarial example ( classified as crane ) , and Figure 1 ( g ) visualizes the difference between the two images . We experimentally show that our attack is competitive with state-of-the-art practical black-box attacks ( AutoZoom by Tu et al . ( 2019 ) and GenAttack by Alzantot et al . ( 2019 ) ) and fools the ensemble adversarial training defense by Tramèr et al . ( 2018 ) . Our results strengthen the claim of Ilyas et al . ( 2019 ) who suggested that weak ( non-robust ) features of the data contribute to the existence of adversarial examples . However , unlike Ilyas et al . ( 2019 ) , we do not focus on features that stem from the DNN . This allows us to expose interesting patterns , expressed via simple functions , that the DNN has learned or missed . PCA features We obtain an initial set of features by running principal component analysis ( PCA ) . PCA provides us with an automatic way to extract useful features from a dataset . For example , the sky color feature of the airplane in Figure 1 ( a ) is in fact the second PCA dimension of this class of images in CIFAR10 . Figure 1 ( h ) shows the effect of perturbing this feature by small constants ( multiplied by δ = 5 ) in the PCA domain and projecting it back to the image domain . Our choice of using PCA is inspired by earlier work . We hypothesized that DNN may learn ( some ) of the PCA features by relying on works that showed that PCA can capture semantic features of the dataset ( e.g. , Zhoua et al . ( 2008 ) ; Jolliffe ( 2002 ) ) and that DNNs learn semantic features ( e.g. , Zeiler & Fergus ( 2014 ) ) . Further , PCA has been linked to adversarial examples : several works showed how to detect adversarial examples using PCA ( Hendrycks & Gimpel. , 2017 ; Jere et al. , 2019 ; Li & Li , 2017 ) , others utilized PCA to approximate an adversarial manifold ( Zhang et al. , 2020 ) , and others constructed an attack that modified only the first PCA dimensions ( Carlini & Wagner. , 2017 ) . Computing the exact PCA dimensions requires perfect-knowledge of the dataset and is time-consuming for large datasets ( e.g. , ImageNet ) . We show that an approximation of the PCA dimensions is sufficient to obtain robust and weak features and that these can be computed from a small subset of the dataset ( not used for training ) . The assumption that the attacker has access to a small dataset of similar distribution to the training set is often valid in practice ( e.g. , for traffic sign recognition benchmarks , like GTSRB , or face recognition applications ) . We evaluate our approach on six datasets and various network architectures . Results indicate that ( 1 ) compared to the standard neighborhoods , our provably robust feature-guided neighborhoods have larger volume , on average by 1.5x and up to 4.5x , and they contain 1079x more images ( average is taken over the exponent ) , and ( 2 ) our adversarial examples are generated using at least 12.2x fewer queries and have at least 2.8x lower L2 distortion compared to state-of-the-art practical black-box attacks . We also show that our attack is effective even against ensemble adversarial training . To conclude , our main contributions are : • Definitions of networks ’ robustness and weakness to feature perturbations ( Section 2 ) . • A sampling algorithm for finding the most robust and weakest features ( Section 3 ) . • Feature-based neighborhoods and their robustness analysis ( Section 4 ) . • Feature-based adversarial examples ( Section 5 ) . • Instantiation with PCA features ( Section 6 ) . • Extensive evaluation of our approach ( Section 7 ) . 2 ROBUST AND WEAK FEATURES . In this section , we define robust and weak features . We focus on classifiers , D : Rd → Rc , given as a black-box , and trained on input-output pairs ( x , z ) sampled from a distribution T . We denote by class ( D ( x ) ) the classification of x by D ( i.e. , the index i maximizing D ( x ) i ) . Features A feature function f : Rd → Rd is a bijective function mapping inputs to real-valued vectors . The assumption that f is bijective simplifies our definitions but these can be extended to other kinds of feature functions . The ith component of a feature function f is called the feature i of f . Feature perturbation For a feature function f : Rd → Rd , index i , and a scalar δ ∈ R , we define the perturbation of x along feature i by δ as follows : xf , i , δ = f−1 ( f ( x ) + δ { i } ) , where δ { i } ∈ Rd is a vector whose entries are zero except for the ith entry which is δ . Network robustness to features perturbations Given a feature function f , an index i , and a scalar δ ∈ R+ , we say the feature i of f is δ-robust if any perturbation to the feature i up to δ does not decrease the class confidence of D on x denoted by CC ( D , x ) ( shortly defined ) . Formally : P ( x , z ) ~T ( [ ∀δ′ ∈ [ 0 , δ ] .class ( D ( xf , i , δ′ ) ) = class ( D ( x ) ) and CC ( D , xf , i , δ′ ) ≥ CC ( D , x ) ] ) = 1 . The expression [ · ] is the Iverson bracket : it is 1 if the formula evaluates to true , and 0 otherwise . We define δ-robustness for δ ∈ R− similarly ( except that δ′ ∈ [ δ , 0 ] ) . If i is δ-robust and δ > 0 , we say the robustness direction of i is + ( positive ) ; otherwise if δ < 0 the direction is − ( negative ) . We define network weakness dually . The feature i of f is δ-weak if any perturbation to the feature i up to δ changes the classification or decreases the class confidence . Formally , for δ ∈ R+ : P ( x , z ) ~T ( [ ∀δ′ ∈ [ 0 , δ ] .class ( D ( xf , i , δ′ ) ) 6= class ( D ( x ) ) or CC ( D , xf , i , δ′ ) ≤ CC ( D , x ) ] ) = 1 . Class confidence The definition of class confidence CC ( D , x ) is orthogonal to our approach . One can define it as the class probability . However , we experimentally observed that it is better to relax the requirement about the absolute value and instead focus on the difference between the probability of the highest ranked class and the second highest , that is : CC ( D , x ) = D ( x ) class ( D ( x ) ) −max { D ( x ) i | i ∈ { 0 , . . . , c− 1 } \ { class ( D ( x ) ) } } . Most robust and weakest features Generally , we do not expect to find robust features if the features are defined orthogonally to the network ’ s structure and parameters . However , if we relax the definition and quantify the robustness level of a feature , we can expose connections between the feature and the network ’ s robustness behavior . We define the robustness level RL as : RL = P ( x , z ) ~T ( [ ∀δ′ ∈ [ 0 , δ ] .class ( D ( xf , i , δ′ ) ) = class ( D ( x ) ) and CC ( D , xf , i , δ′ ) ≥ CC ( D , x ) ] ) . We define the problem of finding the most robust features as finding k features maximizing the robustness level . Dually , we define the weakness level and the problem of finding k weakest features .

Summary: The overall quality of the paper is good. This paper proposed a feature perturbation procedure, as a comparison to the commonly used perturbation to the original input data. Given access to a feature mapping and a black-box classifier, the proposed procedure is able to select the most robust/weak features. This then can be used for two important tasks: to determine a robust neighborhood for a data point using the robust features and to design adversarial examples using the weak features. For the first task, the feature-based robust neighborhood proposed by this paper is shown by experiments to contain far more points than the traditional input-based neighborhood. For the second task, the feature-based adversarial examples require less query to the black-box classifier and have less distortion from the original data points compared with other competitive methods, and thus are more human-imperceptible. These characteristics make the procedure appealing.