Robustness against Relational Adversary

Test-time adversarial attacks have posed serious challenges to the robustness of machine-learning models , and in many settings the adversarial perturbation need not be bounded by small �p-norms . Motivated by the semantics-preserving attacks in vision and security domain , we investigate relational adversaries , a broad class of attackers who create adversarial examples that are in a reflexive-transitive closure of a logical relation . We analyze the conditions for robustness and propose normalize-and-predict – a learning framework with provable robustness guarantee . We compare our approach with adversarial training and derive an unified framework that provides benefits of both approaches . Guided by our theoretical findings , we apply our framework to image classification and malware detection . Results of both tasks show that attacks using relational adversaries frequently fool existing models , but our unified framework can significantly enhance their robustness . 1 INTRODUCTION . The robustness of machine learning ( ML ) systems has been challenged by test-time attacks using adversarial examples ( Szegedy et al. , 2013 ) . These adversarial examples are intentionally manipulated inputs that preserve the essential characteristics of the original inputs , and thus are expected to have the same test outcome as the originals by human standard ; yet they severely affect the performance of many ML models across different domains ( Moosavi-Dezfooli et al. , 2016 ; Eykholt et al. , 2018 ; Qin et al. , 2019 ) . As models in high-stake domains such as system security are also undermined by attacks ( Grosse et al. , 2017 ; Rosenberg et al. , 2018 ; Hu & Tan , 2018 ) , robust ML in adversarial test environment becomes an imperative task for the ML community . Existing work on test-time attacks predominately considers �p-norm bounded adversarial manipulation ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ) . However , in many security-critical settings , the adversarial examples need not respect the �p-norm constraint as long as they preserve the malicious semantics . In malware detection , for example , a malware author can implement the same function using different APIs , or bind a malware within benign softwares like video games or office tools . The modified malware preserves the malicious functionality despite the drastically different syntactic features . Hence , focusing on adversarial examples of small �p-norm in this setting will fail to address a sizable attack surface that attackers can exploit to evade detectors . In addition to security threats , another rising concern on ML models is the spurious correlations they could have learned in a biased data set . Ribeiro et al . ( 2016 ) show that a highly accurate wolf-vshusky-dog classifier indeed bases its prediction on the presence/absence of snow in the background . A reliable model , in contrast , should be robust to changes of this nature . Although dubbed as semantic perturbation or manipulation ( Mohapatra et al. , 2020 ; Bhattad et al. , 2019 ) , these changes do not alter the core of the semantics of input data , thus , we still consider them to be semantics-preserving pertaining to the classification task . Since such semantics-preserving changes often resulted in large �p-norms , they are likely to render the existing �p-norm based defenses ineffective . In this paper , we consider a general attack framework in which attackers create adversarial examples by transforming the original inputs via a set of rules in a semantics-preserving manner . Unlike the prior works ( Rosenberg et al. , 2018 ; Hu & Tan , 2018 ; Hosseini et al. , 2017 ; Hosseini & Poovendran , 2018 ) which investigate specific adversarial settings , our paper extends the scope of attacks to general logical transformation : we unify the threat models into a powerful relational adversary , which can readily incorporate more complex input transformations . From the defense perspective , recent work has started to look beyond �p-norm constraints , including adversarial training ( Grosse et al. , 2017 ; Rosenberg et al. , 2019 ; Lei et al. , 2019 ) , verificationloss regularization ( Huang et al. , 2019 ) and invariance-induced regularization ( Yang et al. , 2019 ) . Adversarial training in principle can achieve high robust accuracy when the adversarial example in the training loop maximizes the loss . However , finding such adversarial examples is in general NPhard ( Katz et al. , 2017 ) , and we show in Sec 4 that it is even PSPACE-hard for semantics-preserving attacks that are considered in this paper . Huang et al . ( 2019 ) and Yang et al . ( 2019 ) add regularizers that incorporate model robustness as part of the training objective . However , such regularization can not be strictly enforced in training , and neither can the model robustness . These limitations still cause vulnerability to semantics-preserving attacks . Normalize-and-Predict Learning Framework This paper attempts to overcome the limitations of prior work by introducing a learning framework that guarantees robustness by design . In particular , we target a relational adversary , whose admissible manipulation is specified by a logical relation . A logical relation is a set of input pairs , each of which consists of a source and target of an atomic , semantics-preserving transformation . We consider a strong adversary who can apply an arbitrary number of transformations . Our paper makes the following contribution towards the theoretical understanding of robust ML against relational adversaries : 1 . We formally describe admissible adversarial manipulation using logical relations , and characterize the necessary and sufficient conditions for robustness to relational adversaries . 2 . We propose normalize-and-predict ( hereinafter abbreviated as N & P ) , a learning framework that first converts each data input to a well-defined and unique normal form and then trains and classifies over the normalized inputs . We show that our framework has guaranteed robustness , and characterize conditions to different levels of robustness-accuracy trade-off . 3 . We compare N & P to the popular adversarial training framework , which directly optimizes for accuracy under attacks . We show that N & P has the advantage in terms of explicit robustness guarantee and reduced training complexity , and in certain cases yields the same model accuracy as adversarial training . Motivated by the comparison , we propose a unified framework , which selectively normalizes over relations that tend to preserve the model accuracy and adversarially trains over the rest . Our unified approach gets the benefits from both frameworks . We then apply our theoretical findings to malware detection and image classification . For the former , first , we formulate two types of common program transformation — ( 1 ) addition of redundant libraries and API calls , and ( 2 ) substitution of equivalent API calls — as logical relations . Next , we instantiate our learning framework to these relations , and propose two generic relational adversarial attacks to determine the robustness of a model . Finally , we perform experiments over Sleipnir , a real-world WIN32 malware data set . Regarding image classification , we reused an attack method proposed by the prior work ( Hosseini & Poovendran , 2018 ) — shifting of the hue in the HSV color space — that can be deemed as a specific instantiation of our attack framework . We then compare the accuracy and robustness of ResNet-32 ( He et al. , 2016 ) , a common image classification model , trained with the unified framework against the standard adversarial training on CIFAR-10 ( Krizhevsky et al. , 2009 ) . The results we obtained in both tasks show that : 1 . Attacks using addition and substitution suffice to evade existing ML malware detectors . 2 . Our unified approach using input normalization and adversarial training achieves highest robust accuracy among all baselines in malware detection . The drop in accuracy on clean inputs is small and the computation cost is lower than pure adversarial training . 3 . When trained with the unified learning framework , ResNet-32 achieves similar clean accuracy but significantly higher robust accuracy than adversarial training alone . Finally , based on our theoretical and empirical results , we conclude that input normalization is vital to robust learning against relational adversaries . We believe techniques that can improve the quality of normalization are promising directions for future work . 2 RELATED WORK .. Test-time attacks using adversarial examples have been extensively studied in the past several years . Research has shown ML models are vulnerable to such attack in a variety of application domains ( Moosavi-Dezfooli et al. , 2016 ; Chen et al. , 2017 ; Papernot et al. , 2017 ; Eykholt et al. , 2018 ; Ebrahimi et al. , 2018 ; Qin et al. , 2019 ; Yang et al. , 2020 ) including system security where reliable defense is absolutely essential . For instance , Grosse et al . ( 2017 ) and Al-Dujaili et al . ( 2018 ) evade API/library usage based malware detectors by adding redundant API calls ; Rosenberg et al . ( 2018 ) , Hu & Tan ( 2018 ) , and Rosenberg et al . ( 2019 ) successfully attack running-time behavior based detectors by adding redundant execution traces ; Pierazzi et al . ( 2020 ) extend the attacks from feature-space to problem-space , propose a framework to describe real-world attacker ’ s constraints and create realistic attack instances using automated software transplantation . On the defense end , the work closest to ours in spirit is Yang et al . ( 2019 ) , which adds invarianceinduced regularizers to the training process . Their work however differs from ours in two major ways . First , their work considers a specific spatial transformation attack in image classification ; our work considers a general adversary based on logic relations . Second , their regularizer may not enforce the model robustness on finite samples as they are primarily interested in enhancing the model accuracy . In contrast , our framework emphasizes robustness which is enforced by design . Grosse et al . ( 2017 ) ; Al-Dujaili et al . ( 2018 ) ; Rosenberg et al . ( 2019 ) improve robustness via adversarial training ; we show such approach is hard to optimize . Incer et al . ( 2018 ) ; Kouzemtchenko ( 2018 ) enforce monotonicity over model outputs so that the addition of feature values always increase the maliciousness score . These approaches are limited to guarding against the addition attacks , thus lacks generality . Last , Xu et al . ( 2017 ) use feature squeezing , which quantizes the feature values in order to reduce the number of adversarial choices . However , their defense is for �p-norm adversaries and thus inapplicable for relational attacks . Normalization is a technique to reduce the number of syntactically distinct instances . First introduced to network security in the early 2000s in the context of intrusion detection systems ( Handley et al. , 2001 ) , it was later applied to malware detection ( Christodorescu et al. , 2007 ; Coogan et al. , 2011 ; Bichsel et al. , 2016 ; Salem & Banescu , 2016 ; Baumann et al. , 2017 ) . Our work addresses the open question whether normalization is useful for ML under relational adversary by investigating its impact on both model robustness and accuracy . 3 BACKGROUND . In this section , we first describe the learning task , then formalize the potential adversarial manipulation as logical relations , and eventually derive the notion of robustness to relational adversaries . Learning Task . We consider a data distribution D over a input space X and categorical label space Y . We use bold face letters , e.g . x , for input vectors and y for the label . Given a hypothesis class H , the learner wants to learn a classifier f : X → Y in H that minimizes the risk over the data distribution . In non-adversarial settings , the learner solves minf∈H E ( x , y ) ∼D � ( f , x , y ) , where � is a loss function . For classification , � ( f , x , y ) = 1 ( f ( x ) �= y ) . Logical Relation . A relation R is a set of input pairs , where each pair ( x , z ) specifies a transformation of input x to output z . We write x →R z iff ( x , z ) ∈ R. We write x →∗R z iff x = z or there exists z0 , z1 , · · · , zk ( k > 0 ) such that x = z0 , zi →R zi+1 ( 0 ≤ i < k ) and zk = z . In other words , →∗R is the reflexive-transitive closure of →R . We describe an example relation as follows : Example 1 ( Hue Shifting ) . Let xh , xs , xv denote the hue , saturation and value components of an image x . In a hue shifting relation R , x →R z iff zh = ( xh+δ ) % 1 where δ is a scalar , zs = xs , zv = xv . Since xh changes in a circle , i.e. , hue of 1 is equal to hue of 0 . Hence , we compute the modulo of the hue component with 1 to map zh within [ 0,1 ] ( Appendix B gives the background of HSV ) . In this paper , we also consider unions of relations . Notice that a finite union R of m relations R1 , · · · , Rm is also a relation , and x →R z iff x →Ri z for any i ∈ { 1 , · · · , m } . Threat Model . A test-time adversary replaces a clean test input x with an adversarially manipulated input A ( x ) , where A ( · ) represents the attack algorithm . We consider an adversary who wants to maximize the classification error rate : E ( x , y ) ∼D 1 ( f ( A ( x ) ) �= y ) . We assume white-box attacks1 , i.e . the adversary has total access to f , including its structures , model parameters and any defense mechanism in place . To maintain the malicious semantics , the adversarial input A ( x ) should belong to a feasible set T ( x ) . In this paper , we focus on T ( x ) that is described by relation . We consider a logical relation R that is known to both the learner and the adversary , and we define a relational adversary as the following . Definition 1 ( relational adversary ) . An adversary is said to be R-relational if T ( x ) = { z |x →∗R z } , i.e . each element in R represents an admissible transformation , and the adversary can apply arbitrary number of transformation specified by R. We can then define the robustness of a classifier f by how often its prediction is consistent under attack , and robust accuracy as the fraction of predictions that are both robust and accurate . Definition 2 ( Robustness and robust accuracy ) . Let Q ( R , f , x ) be the following statement : ∀ z ( ( x →∗R z ) ⇒ f ( x ) = f ( z ) ) . Then , a classifier f is robust at x if Q ( R , f , x ) is true , and the robustness of f to an R-relational adversary is : Ex∼DX 1Q ( R , f , x ) , where 1 ( · ) indicates the truth value of a statement and DX is the marginal distribution over inputs . The robust accuracy of f w.r.t . an R-relational adversary is then : E ( x , y ) ∼D 1Q ( R , f , x ) ∧f ( x ) =y . Notice that the robust accuracy of a classifier is no more than the robustness in value because of the extra requirement of f ( x ) = y . Meanwhile , a classifier with the highest robustness accuracy may not always have the highest robustness and vice versa : an intuitive example is that a constant classifier is always robust but not necessarily robustly accurate . In Sec 4 , we will discuss both objectives and characterize the trade-off between them . 4 N & P – A PROVABLY ROBUST LEARNING FRAMEWORK In this section , we introduce N & P , a learning framework which learns and predicts over normalized training and test inputs . We first identify the necessary and sufficient condition for robustness , and propose a normalization procedure that makes N & P provably robust to R-relational adversaries . Finally , we analyze the performance of N & P : since N & P guarantees robustness , the analysis will focus on robustness-accuracy trade-off and provide an in-depth understanding to causes of such trade-off . 4.1 AN OVERVIEW OF THE N & P FRAMEWORK In N & P , the learner first specifies a normalizer N : X → X . We call N ( x ) the ‘ normal form ’ of input x . The learner then both trains the classifier and predicts the test label over the normal forms instead of the original inputs . Let D denote the training set . In the empirical risk minimization learning scheme , for example , the learner will now solve the following problem min f∈H � ( x , y ) ∈D � ( f , N ( x ) , y ) , ( 1 ) and use the minimizer f∗ as the classifier . During test-time , the model will predict f∗ ( N ( x ) ) . Table 1 compares the N & P learning pipeline to normal risk minimization and adversarial training . 1We consider a strong white-box attacker to avoid interference from security by obscurity , which is shown fragile in various other adversarial settings ( Carlini & Wagner , 2017 ) .

Summary: This paper proposed a new framework, relational adversary, to study adversarial robustness under multiple (sequential) data transformations. The authors also made robustness guarantees based on the Normalizle-and-Predict scenario. Improved robustness is numerically presented on two tasks: malware detection (under add/substitute attack) and image classification (under joint hue and Linfinity perturbation).