Robustness against Relational Adversary

Test-time adversarial attacks have posed serious challenges to the robustness of machine-learning models , and in many settings the adversarial perturbation need not be bounded by small �p-norms . Motivated by the semantics-preserving attacks in vision and security domain , we investigate relational adversaries , a broad class of attackers who create adversarial examples that are in a reflexive-transitive closure of a logical relation . We analyze the conditions for robustness and propose normalize-and-predict – a learning framework with provable robustness guarantee . We compare our approach with adversarial training and derive an unified framework that provides benefits of both approaches . Guided by our theoretical findings , we apply our framework to image classification and malware detection . Results of both tasks show that attacks using relational adversaries frequently fool existing models , but our unified framework can significantly enhance their robustness . 1 INTRODUCTION . The robustness of machine learning ( ML ) systems has been challenged by test-time attacks using adversarial examples ( Szegedy et al. , 2013 ) . These adversarial examples are intentionally manipulated inputs that preserve the essential characteristics of the original inputs , and thus are expected to have the same test outcome as the originals by human standard ; yet they severely affect the performance of many ML models across different domains ( Moosavi-Dezfooli et al. , 2016 ; Eykholt et al. , 2018 ; Qin et al. , 2019 ) . As models in high-stake domains such as system security are also undermined by attacks ( Grosse et al. , 2017 ; Rosenberg et al. , 2018 ; Hu & Tan , 2018 ) , robust ML in adversarial test environment becomes an imperative task for the ML community . Existing work on test-time attacks predominately considers �p-norm bounded adversarial manipulation ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ) . However , in many security-critical settings , the adversarial examples need not respect the �p-norm constraint as long as they preserve the malicious semantics . In malware detection , for example , a malware author can implement the same function using different APIs , or bind a malware within benign softwares like video games or office tools . The modified malware preserves the malicious functionality despite the drastically different syntactic features . Hence , focusing on adversarial examples of small �p-norm in this setting will fail to address a sizable attack surface that attackers can exploit to evade detectors . In addition to security threats , another rising concern on ML models is the spurious correlations they could have learned in a biased data set . Ribeiro et al . ( 2016 ) show that a highly accurate wolf-vshusky-dog classifier indeed bases its prediction on the presence/absence of snow in the background . A reliable model , in contrast , should be robust to changes of this nature . Although dubbed as semantic perturbation or manipulation ( Mohapatra et al. , 2020 ; Bhattad et al. , 2019 ) , these changes do not alter the core of the semantics of input data , thus , we still consider them to be semantics-preserving pertaining to the classification task . Since such semantics-preserving changes often resulted in large �p-norms , they are likely to render the existing �p-norm based defenses ineffective . In this paper , we consider a general attack framework in which attackers create adversarial examples by transforming the original inputs via a set of rules in a semantics-preserving manner . Unlike the prior works ( Rosenberg et al. , 2018 ; Hu & Tan , 2018 ; Hosseini et al. , 2017 ; Hosseini & Poovendran , 2018 ) which investigate specific adversarial settings , our paper extends the scope of attacks to general logical transformation : we unify the threat models into a powerful relational adversary , which can readily incorporate more complex input transformations . From the defense perspective , recent work has started to look beyond �p-norm constraints , including adversarial training ( Grosse et al. , 2017 ; Rosenberg et al. , 2019 ; Lei et al. , 2019 ) , verificationloss regularization ( Huang et al. , 2019 ) and invariance-induced regularization ( Yang et al. , 2019 ) . Adversarial training in principle can achieve high robust accuracy when the adversarial example in the training loop maximizes the loss . However , finding such adversarial examples is in general NPhard ( Katz et al. , 2017 ) , and we show in Sec 4 that it is even PSPACE-hard for semantics-preserving attacks that are considered in this paper . Huang et al . ( 2019 ) and Yang et al . ( 2019 ) add regularizers that incorporate model robustness as part of the training objective . However , such regularization can not be strictly enforced in training , and neither can the model robustness . These limitations still cause vulnerability to semantics-preserving attacks . Normalize-and-Predict Learning Framework This paper attempts to overcome the limitations of prior work by introducing a learning framework that guarantees robustness by design . In particular , we target a relational adversary , whose admissible manipulation is specified by a logical relation . A logical relation is a set of input pairs , each of which consists of a source and target of an atomic , semantics-preserving transformation . We consider a strong adversary who can apply an arbitrary number of transformations . Our paper makes the following contribution towards the theoretical understanding of robust ML against relational adversaries : 1 . We formally describe admissible adversarial manipulation using logical relations , and characterize the necessary and sufficient conditions for robustness to relational adversaries . 2 . We propose normalize-and-predict ( hereinafter abbreviated as N & P ) , a learning framework that first converts each data input to a well-defined and unique normal form and then trains and classifies over the normalized inputs . We show that our framework has guaranteed robustness , and characterize conditions to different levels of robustness-accuracy trade-off . 3 . We compare N & P to the popular adversarial training framework , which directly optimizes for accuracy under attacks . We show that N & P has the advantage in terms of explicit robustness guarantee and reduced training complexity , and in certain cases yields the same model accuracy as adversarial training . Motivated by the comparison , we propose a unified framework , which selectively normalizes over relations that tend to preserve the model accuracy and adversarially trains over the rest . Our unified approach gets the benefits from both frameworks . We then apply our theoretical findings to malware detection and image classification . For the former , first , we formulate two types of common program transformation — ( 1 ) addition of redundant libraries and API calls , and ( 2 ) substitution of equivalent API calls — as logical relations . Next , we instantiate our learning framework to these relations , and propose two generic relational adversarial attacks to determine the robustness of a model . Finally , we perform experiments over Sleipnir , a real-world WIN32 malware data set . Regarding image classification , we reused an attack method proposed by the prior work ( Hosseini & Poovendran , 2018 ) — shifting of the hue in the HSV color space — that can be deemed as a specific instantiation of our attack framework . We then compare the accuracy and robustness of ResNet-32 ( He et al. , 2016 ) , a common image classification model , trained with the unified framework against the standard adversarial training on CIFAR-10 ( Krizhevsky et al. , 2009 ) . The results we obtained in both tasks show that : 1 . Attacks using addition and substitution suffice to evade existing ML malware detectors . 2 . Our unified approach using input normalization and adversarial training achieves highest robust accuracy among all baselines in malware detection . The drop in accuracy on clean inputs is small and the computation cost is lower than pure adversarial training . 3 . When trained with the unified learning framework , ResNet-32 achieves similar clean accuracy but significantly higher robust accuracy than adversarial training alone . Finally , based on our theoretical and empirical results , we conclude that input normalization is vital to robust learning against relational adversaries . We believe techniques that can improve the quality of normalization are promising directions for future work . 2 RELATED WORK .. Test-time attacks using adversarial examples have been extensively studied in the past several years . Research has shown ML models are vulnerable to such attack in a variety of application domains ( Moosavi-Dezfooli et al. , 2016 ; Chen et al. , 2017 ; Papernot et al. , 2017 ; Eykholt et al. , 2018 ; Ebrahimi et al. , 2018 ; Qin et al. , 2019 ; Yang et al. , 2020 ) including system security where reliable defense is absolutely essential . For instance , Grosse et al . ( 2017 ) and Al-Dujaili et al . ( 2018 ) evade API/library usage based malware detectors by adding redundant API calls ; Rosenberg et al . ( 2018 ) , Hu & Tan ( 2018 ) , and Rosenberg et al . ( 2019 ) successfully attack running-time behavior based detectors by adding redundant execution traces ; Pierazzi et al . ( 2020 ) extend the attacks from feature-space to problem-space , propose a framework to describe real-world attacker ’ s constraints and create realistic attack instances using automated software transplantation . On the defense end , the work closest to ours in spirit is Yang et al . ( 2019 ) , which adds invarianceinduced regularizers to the training process . Their work however differs from ours in two major ways . First , their work considers a specific spatial transformation attack in image classification ; our work considers a general adversary based on logic relations . Second , their regularizer may not enforce the model robustness on finite samples as they are primarily interested in enhancing the model accuracy . In contrast , our framework emphasizes robustness which is enforced by design . Grosse et al . ( 2017 ) ; Al-Dujaili et al . ( 2018 ) ; Rosenberg et al . ( 2019 ) improve robustness via adversarial training ; we show such approach is hard to optimize . Incer et al . ( 2018 ) ; Kouzemtchenko ( 2018 ) enforce monotonicity over model outputs so that the addition of feature values always increase the maliciousness score . These approaches are limited to guarding against the addition attacks , thus lacks generality . Last , Xu et al . ( 2017 ) use feature squeezing , which quantizes the feature values in order to reduce the number of adversarial choices . However , their defense is for �p-norm adversaries and thus inapplicable for relational attacks . Normalization is a technique to reduce the number of syntactically distinct instances . First introduced to network security in the early 2000s in the context of intrusion detection systems ( Handley et al. , 2001 ) , it was later applied to malware detection ( Christodorescu et al. , 2007 ; Coogan et al. , 2011 ; Bichsel et al. , 2016 ; Salem & Banescu , 2016 ; Baumann et al. , 2017 ) . Our work addresses the open question whether normalization is useful for ML under relational adversary by investigating its impact on both model robustness and accuracy . 3 BACKGROUND . In this section , we first describe the learning task , then formalize the potential adversarial manipulation as logical relations , and eventually derive the notion of robustness to relational adversaries . Learning Task . We consider a data distribution D over a input space X and categorical label space Y . We use bold face letters , e.g . x , for input vectors and y for the label . Given a hypothesis class H , the learner wants to learn a classifier f : X → Y in H that minimizes the risk over the data distribution . In non-adversarial settings , the learner solves minf∈H E ( x , y ) ∼D � ( f , x , y ) , where � is a loss function . For classification , � ( f , x , y ) = 1 ( f ( x ) �= y ) . Logical Relation . A relation R is a set of input pairs , where each pair ( x , z ) specifies a transformation of input x to output z . We write x →R z iff ( x , z ) ∈ R. We write x →∗R z iff x = z or there exists z0 , z1 , · · · , zk ( k > 0 ) such that x = z0 , zi →R zi+1 ( 0 ≤ i < k ) and zk = z . In other words , →∗R is the reflexive-transitive closure of →R . We describe an example relation as follows : Example 1 ( Hue Shifting ) . Let xh , xs , xv denote the hue , saturation and value components of an image x . In a hue shifting relation R , x →R z iff zh = ( xh+δ ) % 1 where δ is a scalar , zs = xs , zv = xv . Since xh changes in a circle , i.e. , hue of 1 is equal to hue of 0 . Hence , we compute the modulo of the hue component with 1 to map zh within [ 0,1 ] ( Appendix B gives the background of HSV ) . In this paper , we also consider unions of relations . Notice that a finite union R of m relations R1 , · · · , Rm is also a relation , and x →R z iff x →Ri z for any i ∈ { 1 , · · · , m } . Threat Model . A test-time adversary replaces a clean test input x with an adversarially manipulated input A ( x ) , where A ( · ) represents the attack algorithm . We consider an adversary who wants to maximize the classification error rate : E ( x , y ) ∼D 1 ( f ( A ( x ) ) �= y ) . We assume white-box attacks1 , i.e . the adversary has total access to f , including its structures , model parameters and any defense mechanism in place . To maintain the malicious semantics , the adversarial input A ( x ) should belong to a feasible set T ( x ) . In this paper , we focus on T ( x ) that is described by relation . We consider a logical relation R that is known to both the learner and the adversary , and we define a relational adversary as the following . Definition 1 ( relational adversary ) . An adversary is said to be R-relational if T ( x ) = { z |x →∗R z } , i.e . each element in R represents an admissible transformation , and the adversary can apply arbitrary number of transformation specified by R. We can then define the robustness of a classifier f by how often its prediction is consistent under attack , and robust accuracy as the fraction of predictions that are both robust and accurate . Definition 2 ( Robustness and robust accuracy ) . Let Q ( R , f , x ) be the following statement : ∀ z ( ( x →∗R z ) ⇒ f ( x ) = f ( z ) ) . Then , a classifier f is robust at x if Q ( R , f , x ) is true , and the robustness of f to an R-relational adversary is : Ex∼DX 1Q ( R , f , x ) , where 1 ( · ) indicates the truth value of a statement and DX is the marginal distribution over inputs . The robust accuracy of f w.r.t . an R-relational adversary is then : E ( x , y ) ∼D 1Q ( R , f , x ) ∧f ( x ) =y . Notice that the robust accuracy of a classifier is no more than the robustness in value because of the extra requirement of f ( x ) = y . Meanwhile , a classifier with the highest robustness accuracy may not always have the highest robustness and vice versa : an intuitive example is that a constant classifier is always robust but not necessarily robustly accurate . In Sec 4 , we will discuss both objectives and characterize the trade-off between them . 4 N & P – A PROVABLY ROBUST LEARNING FRAMEWORK In this section , we introduce N & P , a learning framework which learns and predicts over normalized training and test inputs . We first identify the necessary and sufficient condition for robustness , and propose a normalization procedure that makes N & P provably robust to R-relational adversaries . Finally , we analyze the performance of N & P : since N & P guarantees robustness , the analysis will focus on robustness-accuracy trade-off and provide an in-depth understanding to causes of such trade-off . 4.1 AN OVERVIEW OF THE N & P FRAMEWORK In N & P , the learner first specifies a normalizer N : X → X . We call N ( x ) the ‘ normal form ’ of input x . The learner then both trains the classifier and predicts the test label over the normal forms instead of the original inputs . Let D denote the training set . In the empirical risk minimization learning scheme , for example , the learner will now solve the following problem min f∈H � ( x , y ) ∈D � ( f , N ( x ) , y ) , ( 1 ) and use the minimizer f∗ as the classifier . During test-time , the model will predict f∗ ( N ( x ) ) . Table 1 compares the N & P learning pipeline to normal risk minimization and adversarial training . 1We consider a strong white-box attacker to avoid interference from security by obscurity , which is shown fragile in various other adversarial settings ( Carlini & Wagner , 2017 ) .

Summary: This work investigates relational adversary which creates adversarial examples by transforming the original inputs via a logical relation. The authors analyze the conditions for robustness to relational adversary and propose a learning framework which learns and predicts over normalized training and test inputs. Since their framework may incur excessive loss in accuracy, the authors then propose a unified framework by combining input normalization with adversarial training and claim that their unified framework can preserve the optimal robust accuracy. Empirical results show that their unified framework can enhance model robustness on two real-world tasks.