Towards Understanding the Robustness Against Evasion Attack on Categorical Data

1 INTRODUCTION . Categorical data pervasively exist in real-world safety-critical Machine-Learning-as-a-Service ( MLaaS ) applications , such as ML-driven intrusion detection and digital healthcare . The vulnerability to attacks by intentionally crafting categorical signatures raises concerns on trust and utility of the ML-based analytic services . Characterizing and assessing adversarial robustness on categorical data can thus help evaluate the reliability of the core ML models and flag potential evading efforts . For a classifier 5 with categorical inputs G , the adversarial risk of 5 under evasion attack can be formulated as follows . Definition 1 5 : G→ H denotes a classifier with categorical inputs G. Let ` G , H denote the joint distribution of ( G , H ) . The expected adversarial risk of the classifier 5 under evasion attack is formulated as : R03EY = E ( G , H ) ∼ ` G , H sup |diff ( G , Ĝ ) |≤Y ℓ ( 5 ( Ĝ ) , H ) ( 1 ) where ℓ is the misclassification loss function , G and Ĝ are an unperturbed and the generated adversarial sample respectively . Y denotes the attack budget of evasion attack . G is correctly classified ( 5 ( G ) = H ) . Intuitively , a classifier 5 is more robust against adversarial perturbations , if its adversarial risk is low given the attack budget limiting the number of changed categorical features . Unlike continuous data , a categorical variable can be valued with only one categorical value among others . These categorical values have no intrinsic ordering to the categories . Evasion attack manipulating categorical inputs is in nature an NP-hard knapsack problem . The discontinuous nature raises two fundamental yet rarely addressed questions to evaluate the adversarial risk on categorical data in practice : • Q1 What are the key factors determining 5 ’ s adversarial risk R03EY on categorical data ? • Q2 For a general classifier 5 , can we assess the adversarial risk of 5 with categorical inputs with provably accuracy guarantee ? Despite recent efforts of adversarial vulnerability exploration with discrete data , both questions remain open for several reasons . First , the discontinuity of categorical space prevents the direct use of the previous progress on adversarial risk analysis with continuous data . The local subspace assumption of ; ? -bounded adversarial attacks does not apply to the categorical features ( Hein & Andriushchenko , 2017 ; Wang et al. , 2018 ; Fawzi et al. , 2016 ; Gilmer et al. , 2018 ; Yin et al. , 2019 ; Khim & Loh , 2018 ; Tu et al. , 2019 ) . Second , most practices of discrete adversarial attack are domain specific and depend heavily on domain knowledge . ( Bojchevski & Günnemann , 2019 ; Bojchevski & Günnemann , 2019 ; Zugner & Gunnemann , 2019 ) focus on building differentiable surrogate functions to Graph Neural Networks to facilitate searching for feasible poisoning edits over graph structures and node attributes . ( Narodytska & Kasiviswanathan , 2017 ; Croce & Hein , 2019 ) conduct ! 0-norm perturbations only within local image areas containing sensitive features for image classification . ( Qi et al. , 2019 ; Wang et al. , 2020a ) require non-negativity on the parameters of deep neural networks to deliver provably accurate greedy attacks via submodular function optimization . The non-negativity constraint is unnatural for real-world ML practices . It does harm the utility of the classifier . For a more general classifier with categorical inputs , a provably optimal and domain-agnostic method for attack and adversarial risk evaluation is yet to establish . Using greedy search or the well-known Branch-and-Bound method on a general knapsack problem provides no optimality guarantee of the solutions , thus can produce arbitrarily bad results . Our study aims to address these two questions mentioned above from both theoretical and empirical perspectives . First , we derive an information-theoretic characterization of the adversarial risk of a classifier . It unveils that the informativeness of the input categorical instance , the sensitivity of the perturbed categorical features and the information geometry property of the targeted classifier are the three key factors jointly determining adversarial vulnerability of the classifier . Second , our study adopts an assess-by-attack strategy . We show that assessing adversarial robustness of any measurable classifier with categorical inputs can be cast to a weakly submodular maximization problem , with a mild smoothness condition . It can thus be solved using a simple yet efficient greedy attack strategy with provable approximation guarantees . The theoretical findings not only explain the empirical success of the greedy search strategy to generate adversarial textual and image samples ( Gong et al. , 2018 ; Yang et al. , 2018 ; Narodytska & Kasiviswanathan , 2017 ) , but also pave the way to a domainagnostic adversarial robustness assessment on categorical data with provable optimality guarantees . Third , we instantiate the domain-agnostic adversarial risk characterization and assessment with a widely used DNN classifier , i.e . Long Short-Term Memory ( LSTM ) and three different categorical datasets in Section.4 . These datasets are collected from various real-world applications . The experimental results confirm the impact of the 3 risk factors over the adversarial vulnerability of the DNN classifier . 2 RELATED WORK . Tremendous efforts have been made to vulnerability measurement of a classifier under evasion attack ( Hein & Andriushchenko , 2017 ; Wang et al. , 2018 ; Fawzi et al. , 2016 ; Gilmer et al. , 2018 ; Weng et al. , 2019 ; Sinha et al. , 2018 ; Cohen et al. , 2019 ; Shi et al. , 2020 ; Yin et al. , 2019 ; Khim & Loh , 2018 ; Tu et al. , 2019 ) . Most of the previous works focus on evaluating robustness against ; ? -norm perturbations on continuous data . They all assume adversarial samples locate within a smooth ; ? -ball around an input instance , which doesn ’ t hold for categorical data . In contrast , Tu et al . ( 2019 ) covers both numerical and categorical data . It bounds the adversarial risk with a local worst-case risk over a ? -wassernstein ball centered at the training data distribution . This work associates the adversarial risk of a classifier with its rademacher complexity . Nevertheless , we argue that the adversarial vulnerability of a classifier is determined by not only the characteristics of the classifier , e.g . model complexity , but also the properties of the training/testing data instances . Pioneering works of evasion attacks with categorical inputs depend on domain-specific knowledge to facilitate the attack exploration . Kuleshov et al . ( 2018 ) ; Papernot et al . ( 2016 ) ; Miyato et al . ( 2016 ) ; Samanta & Mehta ( 2017 ) ; Gao et al . ( 2018 ) ; Yang et al . ( 2018 ) ; Gong et al . ( 2018 ) ; Ebrahimi et al . ( 2018 ) ; Narodytska & Kasiviswanathan ( 2017 ) ; Croce & Hein ( 2019 ) focus on replacing individual words/phrases to cheat text classifiers , or modifying pixel intensities to bias image classification results . These methods use heuristic semantic rules , e.g. , replacing words with manually defined candidate synonyms and constraining the word change to preserve readability and semantic integrity . Narodytska & Kasiviswanathan ( 2017 ) ; Croce & Hein ( 2019 ) narrow down the search range to the pixels with high pixel-wise sensitivity for image classification . Despite of the sounding empirical results , there is no guarantee on a successful attack within the attack budget . Bojchevski & Günnemann ( 2019 ) ; Bojchevski & Günnemann ( 2019 ) ; D.Zugner et al . ( 2019 ) ; D. Zügner & Günnemann ( 2018 ) ; Akbarnejad & Günnemann ( 2019 ) adopt edge-flipping and node attribute perturbation to poison graph data mining pipelines , e.g. , graph neural networks and graph embedding models . The key idea is to introduce relaxed surrogate functions to the combinatorial attack objective and then solve the relaxed optimization problem instead . Notably , D.Zugner et al . ( 2019 ) ; D. Zügner & Günnemann ( 2018 ) ; Akbarnejad & Günnemann ( 2019 ) define the attack objective as a sum of smallest eigenvalues of the adjacency matrix of a given graph . Though it is not explicitly claimed , it is intrinsically a submodular maximization problem . Qi et al . ( 2019 ) ; Wang et al . ( 2020a ) unveil that simple greedy search can deliver provably effective attacks against DNN classifiers without domain-specific heuristics , if all the link weights between neurons are non-negative . Non-negativity of the parameter guarantees strict submodularity of the attack objective . However , it brings significantly deterioration of the classifier ’ s accuracy , which is not applicable for real-world learning tasks . 3 ROBUSTNESS CHARACTERIZATION AND ASSESSMENT . We assume an input instance G = { G1 , G2 , G3 , ... , G= } of = categorical attributes . Each G8 takes any of < ( < ≥ 1 ) categorical values . The classifier 5 outputs decision probabilities 5H : ( : = 1,2,3 , ... , ) with respect to different class labels . In practices , each categorical value of G 9 8 is cast to a -dimensional pre- trained embedding vector , e.g. , 4 9 8 ∈R , 9 =1,2 , ... , < . To represent an instance G with the embedding vectors of its category values , we define binary variables 1= { 1 9 8 } , 8=1,2 , ... , = , 9=1,2 , ... , < , where 1 9 8 =1 when the 9-th attribute value is present for G8 and 1 9 8 =0 otherwise . An instance G can then be represented by anR=∗ < ∗ tensor with G { 8 , 9 , : } =1 9 8 4 9 8 . Let 1̂= { 1̂ 9 8 } indicate the adversarial modifications introduced into 1 . For a perturbed Ĝ , its 1̂ ≠ 1 . Depending on the type of attacks to implement , e.g. , insertion , deletion or substitution , 1̂ differs from 1 in different ways . Without loss of generality , let H denote the true class label of G and all the other H : ( : = { 1 , ... , −1 } ) are the potential targets of an evasion attack . The goal of attack is to make 5H ( G,1̂ ) as low as possible and increase 5H : ( G,1̂ ) of a fixed : ( targeted attack ) or any : of { 1 , ... , −1 } ( non-targeted attack ) as high as possible simultaneously . We focus on non-targeted evasion attack , and leave the targeted scenario for future study . 3.1 INFORMATION-THEORETIC CHARACTERIZATION OF ADVERSARIAL VULNERABILITY . Theorem 1 For any input instance ( G , H ) and a training set ( sampled from the same underlying distribution ` G , H , 5 ∈ H is trained using ( with a deterministic training paradigm . The expected adversarial risk R03EY defined in Eq.1 can be bounded as in Eq.2 , if the loss function ℓ in R03EY adopts the zero-one loss . R03EY ≥1− 2 ( G ; H , ( ) −2 [ G− ( 5H ; ( ) /2+const log ( 2 ) [ G = BD ? |diff ( 1,1̂ ) |≤Y ( G ; H , ( ) − ( Ĝ ; H , ( ) ( 3 ) where const denotes a constant term . diff ( 1,1̂ ) indicates the set of the categorical attributes modified in the attack . ( G ; H , ( ) denotes the mutual information between the feature G and the pair of label H and training set ( . [ G is the supremum of the difference between the mutual information before and after adversarial perturbation , noted by ( G ; H , ( ) and ( Ĝ ; H , ( ) respectively . ( 5H ; ( ) denotes the mutual information between ( and 5 . Theorem 1 unveils the three impact factors jointly determining adversarial vulnerability of the targeted classifier 5 ( for answering Q1 in Section 1 ) . The proof is given in Appendix.A . In Eq.2 , a lower mutual information ( G ; H , ( ) indicates higher adversarial risk , i.e. , R03EY has a higher lower bound . A lower ( G ; H , ( ) denotes weaker consistency between the input instance ( G , H ) and the training set ( . The classifier trained by ( thus produces weaker decision confidence for such ( G , H ) . With ( G , H ) dropped to the ambiguous zone near the classification boundary , the classifier is more prone to adversarial perturbation . A higher ( 5H ; ( ) leads to a higher adversarial risk according to Eq.2 . The mutual information ( 5H ; ( ) reflects the dependence between the classifier 5 and the training set ( , which can be considered as a lower bound of the VC-dimension for countable hypothesis space 5 ∈H ( Xu & Raginsky , 2017 ; Zhu et al. , 2020 ) . A higher ( 5H ; ( ) thus denotes that 5 is more likely to suffer from being overfitted to ( . Model overfiting is one of the causes of adversarial vulnerability ( Tu et al. , 2019 ) . Resonating with the association between ( 5H ; ( ) and adversarial risk , three popularly used robustness enhancement methods controlling the classifier ’ s complexity and overfiting risk can potentially adjust the adversarial vulnerability of the classifier with categorical inputs . 1 ) Adversarial training ( Miyato et al. , 2016 ; Sinha et al. , 2018 ; Florian et al. , 2018 ; Wang et al. , 2019 ; Shafahi et al. , 2019 ) . The adversarially retrained classifier 5̂ is less correlated with the original training set ( compared to 5 , which reduces ( 5H ; ( ) and may help mitigate the adversarial threat . 2 ) The addition of nuclear norm regularization over the classifier ’ s parameters in the training process ( Ravi et al. , 2019 ; Tu et al. , 2019 ) . The resultant regularized classifier has a controlled model complexity , which can potentially reduce the adversarial risk . 3 ) Random smoothing ( Lee et al. , 2019 ; Levine & Feizi , 2020 ; Dvijotham et al. , 2020 ; Boj . et al. , 2020 ) . Following ( Cohen et al. , 2019 ) , this defense method randomly selects and flips the input categorical features of the targeted classifier . The randomly perturbed classification output is less correlated with the training data ( ’ s distribution , which may reduce adversarial risk .

Summary: The paper mainly studies the adversarial vulnerability of a classifier with categorical inputs. It theoretically analyzes the key factors that determine the robustness of discrete input data and uses greedy strategy to solve the problem. In the experimental part, the importance of the key factors pointed out by the theory is verified, and the robustness scoring is used to conduct experiments under three defense measures.