Towards Understanding the Robustness Against Evasion Attack on Categorical Data

1 INTRODUCTION . Categorical data pervasively exist in real-world safety-critical Machine-Learning-as-a-Service ( MLaaS ) applications , such as ML-driven intrusion detection and digital healthcare . The vulnerability to attacks by intentionally crafting categorical signatures raises concerns on trust and utility of the ML-based analytic services . Characterizing and assessing adversarial robustness on categorical data can thus help evaluate the reliability of the core ML models and flag potential evading efforts . For a classifier 5 with categorical inputs G , the adversarial risk of 5 under evasion attack can be formulated as follows . Definition 1 5 : G→ H denotes a classifier with categorical inputs G. Let ` G , H denote the joint distribution of ( G , H ) . The expected adversarial risk of the classifier 5 under evasion attack is formulated as : R03EY = E ( G , H ) ∼ ` G , H sup |diff ( G , Ĝ ) |≤Y ℓ ( 5 ( Ĝ ) , H ) ( 1 ) where ℓ is the misclassification loss function , G and Ĝ are an unperturbed and the generated adversarial sample respectively . Y denotes the attack budget of evasion attack . G is correctly classified ( 5 ( G ) = H ) . Intuitively , a classifier 5 is more robust against adversarial perturbations , if its adversarial risk is low given the attack budget limiting the number of changed categorical features . Unlike continuous data , a categorical variable can be valued with only one categorical value among others . These categorical values have no intrinsic ordering to the categories . Evasion attack manipulating categorical inputs is in nature an NP-hard knapsack problem . The discontinuous nature raises two fundamental yet rarely addressed questions to evaluate the adversarial risk on categorical data in practice : • Q1 What are the key factors determining 5 ’ s adversarial risk R03EY on categorical data ? • Q2 For a general classifier 5 , can we assess the adversarial risk of 5 with categorical inputs with provably accuracy guarantee ? Despite recent efforts of adversarial vulnerability exploration with discrete data , both questions remain open for several reasons . First , the discontinuity of categorical space prevents the direct use of the previous progress on adversarial risk analysis with continuous data . The local subspace assumption of ; ? -bounded adversarial attacks does not apply to the categorical features ( Hein & Andriushchenko , 2017 ; Wang et al. , 2018 ; Fawzi et al. , 2016 ; Gilmer et al. , 2018 ; Yin et al. , 2019 ; Khim & Loh , 2018 ; Tu et al. , 2019 ) . Second , most practices of discrete adversarial attack are domain specific and depend heavily on domain knowledge . ( Bojchevski & Günnemann , 2019 ; Bojchevski & Günnemann , 2019 ; Zugner & Gunnemann , 2019 ) focus on building differentiable surrogate functions to Graph Neural Networks to facilitate searching for feasible poisoning edits over graph structures and node attributes . ( Narodytska & Kasiviswanathan , 2017 ; Croce & Hein , 2019 ) conduct ! 0-norm perturbations only within local image areas containing sensitive features for image classification . ( Qi et al. , 2019 ; Wang et al. , 2020a ) require non-negativity on the parameters of deep neural networks to deliver provably accurate greedy attacks via submodular function optimization . The non-negativity constraint is unnatural for real-world ML practices . It does harm the utility of the classifier . For a more general classifier with categorical inputs , a provably optimal and domain-agnostic method for attack and adversarial risk evaluation is yet to establish . Using greedy search or the well-known Branch-and-Bound method on a general knapsack problem provides no optimality guarantee of the solutions , thus can produce arbitrarily bad results . Our study aims to address these two questions mentioned above from both theoretical and empirical perspectives . First , we derive an information-theoretic characterization of the adversarial risk of a classifier . It unveils that the informativeness of the input categorical instance , the sensitivity of the perturbed categorical features and the information geometry property of the targeted classifier are the three key factors jointly determining adversarial vulnerability of the classifier . Second , our study adopts an assess-by-attack strategy . We show that assessing adversarial robustness of any measurable classifier with categorical inputs can be cast to a weakly submodular maximization problem , with a mild smoothness condition . It can thus be solved using a simple yet efficient greedy attack strategy with provable approximation guarantees . The theoretical findings not only explain the empirical success of the greedy search strategy to generate adversarial textual and image samples ( Gong et al. , 2018 ; Yang et al. , 2018 ; Narodytska & Kasiviswanathan , 2017 ) , but also pave the way to a domainagnostic adversarial robustness assessment on categorical data with provable optimality guarantees . Third , we instantiate the domain-agnostic adversarial risk characterization and assessment with a widely used DNN classifier , i.e . Long Short-Term Memory ( LSTM ) and three different categorical datasets in Section.4 . These datasets are collected from various real-world applications . The experimental results confirm the impact of the 3 risk factors over the adversarial vulnerability of the DNN classifier . 2 RELATED WORK . Tremendous efforts have been made to vulnerability measurement of a classifier under evasion attack ( Hein & Andriushchenko , 2017 ; Wang et al. , 2018 ; Fawzi et al. , 2016 ; Gilmer et al. , 2018 ; Weng et al. , 2019 ; Sinha et al. , 2018 ; Cohen et al. , 2019 ; Shi et al. , 2020 ; Yin et al. , 2019 ; Khim & Loh , 2018 ; Tu et al. , 2019 ) . Most of the previous works focus on evaluating robustness against ; ? -norm perturbations on continuous data . They all assume adversarial samples locate within a smooth ; ? -ball around an input instance , which doesn ’ t hold for categorical data . In contrast , Tu et al . ( 2019 ) covers both numerical and categorical data . It bounds the adversarial risk with a local worst-case risk over a ? -wassernstein ball centered at the training data distribution . This work associates the adversarial risk of a classifier with its rademacher complexity . Nevertheless , we argue that the adversarial vulnerability of a classifier is determined by not only the characteristics of the classifier , e.g . model complexity , but also the properties of the training/testing data instances . Pioneering works of evasion attacks with categorical inputs depend on domain-specific knowledge to facilitate the attack exploration . Kuleshov et al . ( 2018 ) ; Papernot et al . ( 2016 ) ; Miyato et al . ( 2016 ) ; Samanta & Mehta ( 2017 ) ; Gao et al . ( 2018 ) ; Yang et al . ( 2018 ) ; Gong et al . ( 2018 ) ; Ebrahimi et al . ( 2018 ) ; Narodytska & Kasiviswanathan ( 2017 ) ; Croce & Hein ( 2019 ) focus on replacing individual words/phrases to cheat text classifiers , or modifying pixel intensities to bias image classification results . These methods use heuristic semantic rules , e.g. , replacing words with manually defined candidate synonyms and constraining the word change to preserve readability and semantic integrity . Narodytska & Kasiviswanathan ( 2017 ) ; Croce & Hein ( 2019 ) narrow down the search range to the pixels with high pixel-wise sensitivity for image classification . Despite of the sounding empirical results , there is no guarantee on a successful attack within the attack budget . Bojchevski & Günnemann ( 2019 ) ; Bojchevski & Günnemann ( 2019 ) ; D.Zugner et al . ( 2019 ) ; D. Zügner & Günnemann ( 2018 ) ; Akbarnejad & Günnemann ( 2019 ) adopt edge-flipping and node attribute perturbation to poison graph data mining pipelines , e.g. , graph neural networks and graph embedding models . The key idea is to introduce relaxed surrogate functions to the combinatorial attack objective and then solve the relaxed optimization problem instead . Notably , D.Zugner et al . ( 2019 ) ; D. Zügner & Günnemann ( 2018 ) ; Akbarnejad & Günnemann ( 2019 ) define the attack objective as a sum of smallest eigenvalues of the adjacency matrix of a given graph . Though it is not explicitly claimed , it is intrinsically a submodular maximization problem . Qi et al . ( 2019 ) ; Wang et al . ( 2020a ) unveil that simple greedy search can deliver provably effective attacks against DNN classifiers without domain-specific heuristics , if all the link weights between neurons are non-negative . Non-negativity of the parameter guarantees strict submodularity of the attack objective . However , it brings significantly deterioration of the classifier ’ s accuracy , which is not applicable for real-world learning tasks . 3 ROBUSTNESS CHARACTERIZATION AND ASSESSMENT . We assume an input instance G = { G1 , G2 , G3 , ... , G= } of = categorical attributes . Each G8 takes any of < ( < ≥ 1 ) categorical values . The classifier 5 outputs decision probabilities 5H : ( : = 1,2,3 , ... , ) with respect to different class labels . In practices , each categorical value of G 9 8 is cast to a -dimensional pre- trained embedding vector , e.g. , 4 9 8 ∈R , 9 =1,2 , ... , < . To represent an instance G with the embedding vectors of its category values , we define binary variables 1= { 1 9 8 } , 8=1,2 , ... , = , 9=1,2 , ... , < , where 1 9 8 =1 when the 9-th attribute value is present for G8 and 1 9 8 =0 otherwise . An instance G can then be represented by anR=∗ < ∗ tensor with G { 8 , 9 , : } =1 9 8 4 9 8 . Let 1̂= { 1̂ 9 8 } indicate the adversarial modifications introduced into 1 . For a perturbed Ĝ , its 1̂ ≠ 1 . Depending on the type of attacks to implement , e.g. , insertion , deletion or substitution , 1̂ differs from 1 in different ways . Without loss of generality , let H denote the true class label of G and all the other H : ( : = { 1 , ... , −1 } ) are the potential targets of an evasion attack . The goal of attack is to make 5H ( G,1̂ ) as low as possible and increase 5H : ( G,1̂ ) of a fixed : ( targeted attack ) or any : of { 1 , ... , −1 } ( non-targeted attack ) as high as possible simultaneously . We focus on non-targeted evasion attack , and leave the targeted scenario for future study . 3.1 INFORMATION-THEORETIC CHARACTERIZATION OF ADVERSARIAL VULNERABILITY . Theorem 1 For any input instance ( G , H ) and a training set ( sampled from the same underlying distribution ` G , H , 5 ∈ H is trained using ( with a deterministic training paradigm . The expected adversarial risk R03EY defined in Eq.1 can be bounded as in Eq.2 , if the loss function ℓ in R03EY adopts the zero-one loss . R03EY ≥1− 2 ( G ; H , ( ) −2 [ G− ( 5H ; ( ) /2+const log ( 2 ) [ G = BD ? |diff ( 1,1̂ ) |≤Y ( G ; H , ( ) − ( Ĝ ; H , ( ) ( 3 ) where const denotes a constant term . diff ( 1,1̂ ) indicates the set of the categorical attributes modified in the attack . ( G ; H , ( ) denotes the mutual information between the feature G and the pair of label H and training set ( . [ G is the supremum of the difference between the mutual information before and after adversarial perturbation , noted by ( G ; H , ( ) and ( Ĝ ; H , ( ) respectively . ( 5H ; ( ) denotes the mutual information between ( and 5 . Theorem 1 unveils the three impact factors jointly determining adversarial vulnerability of the targeted classifier 5 ( for answering Q1 in Section 1 ) . The proof is given in Appendix.A . In Eq.2 , a lower mutual information ( G ; H , ( ) indicates higher adversarial risk , i.e. , R03EY has a higher lower bound . A lower ( G ; H , ( ) denotes weaker consistency between the input instance ( G , H ) and the training set ( . The classifier trained by ( thus produces weaker decision confidence for such ( G , H ) . With ( G , H ) dropped to the ambiguous zone near the classification boundary , the classifier is more prone to adversarial perturbation . A higher ( 5H ; ( ) leads to a higher adversarial risk according to Eq.2 . The mutual information ( 5H ; ( ) reflects the dependence between the classifier 5 and the training set ( , which can be considered as a lower bound of the VC-dimension for countable hypothesis space 5 ∈H ( Xu & Raginsky , 2017 ; Zhu et al. , 2020 ) . A higher ( 5H ; ( ) thus denotes that 5 is more likely to suffer from being overfitted to ( . Model overfiting is one of the causes of adversarial vulnerability ( Tu et al. , 2019 ) . Resonating with the association between ( 5H ; ( ) and adversarial risk , three popularly used robustness enhancement methods controlling the classifier ’ s complexity and overfiting risk can potentially adjust the adversarial vulnerability of the classifier with categorical inputs . 1 ) Adversarial training ( Miyato et al. , 2016 ; Sinha et al. , 2018 ; Florian et al. , 2018 ; Wang et al. , 2019 ; Shafahi et al. , 2019 ) . The adversarially retrained classifier 5̂ is less correlated with the original training set ( compared to 5 , which reduces ( 5H ; ( ) and may help mitigate the adversarial threat . 2 ) The addition of nuclear norm regularization over the classifier ’ s parameters in the training process ( Ravi et al. , 2019 ; Tu et al. , 2019 ) . The resultant regularized classifier has a controlled model complexity , which can potentially reduce the adversarial risk . 3 ) Random smoothing ( Lee et al. , 2019 ; Levine & Feizi , 2020 ; Dvijotham et al. , 2020 ; Boj . et al. , 2020 ) . Following ( Cohen et al. , 2019 ) , this defense method randomly selects and flips the input categorical features of the targeted classifier . The randomly perturbed classification output is less correlated with the training data ( ’ s distribution , which may reduce adversarial risk .

Summary: This paper studies the problem of assessing the adversarial robustness of a classifier with categorical inputs, instead of continuous inputs as in the literature. The authors claim that there exist provably optimality guarantees for Lipschitz-continuous classifiers, and proposed an impact factor based on an n information-theoretic analysis. Experimental studies are conducted to support the claims.