Generating Adversarial Computer Programs using Optimized Obfuscations

1 INTRODUCTION . Machine learning ( ML ) models are increasingly being used for software engineering tasks . Applications such as refactoring programs , auto-completing them in editors , and synthesizing GUI code have benefited from ML models trained on large repositories of programs , sourced from popular websites like GitHub ( Allamanis et al. , 2018 ) . They have also been adopted to reason about and assess programs ( Srikant & Aggarwal , 2014 ; Si et al. , 2018 ) , find and fix bugs ( Gupta et al. , 2017 ; Pradel & Sen , 2018 ) , detect malware and vulnerabilities in them ( Li et al. , 2018 ; Zhou et al. , 2019 ) etc . thus complementing traditional program analysis tools . As these models continue to be adopted for such applications , it is important to understand how robust they are to adversarial attacks . Such attacks can have adverse consequences , particularly in settings such as security ( Zhou et al. , 2019 ) and compliance automation ( Pedersen , 2010 ) . For example , an attacker could craft changes in malicious programs in a way which forces a model to incorrectly classify them as being benign , or make changes to pass off code which is licensed as open-source in an organization ’ s proprietary code-base . Adversarially perturbing a program should achieve two goals – a trained model should flip its decision when provided with the perturbed version of the program , and second , the perturbation should be imperceivable . Adversarial attacks have mainly been considered in image classification ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) , where calculated minor changes made to pixels of an image are enough to satisfy the imperceptibility requirement . Such changes escape a human ’ s attention by making the image look the same as before perturbing it , while modifying the underlying representation enough to flip a classifier ’ s decision . However , programs demand a stricter imperceptibility requirement – not only should the changes avoid human attention , but the changed program should also importantly functionally behave the same as the unperturbed program . 1Source code : https : //github.com/ALFA-group/adversarial-code-generation Program obfuscations provide the agency to implement one such set of imperceivable changes in programs . Obfuscating computer programs have long been used as a way to avoid attempts at reverse-engineering them . They transform a program in a way that only hampers humans ’ comprehension of parts of the program , while retaining its original semantics and functionality . For example , one common obfuscation operation is to rename variables in an attempt to hide the program ’ s intent from a reader . Renaming a variable sum in the program statement int sum = 0 to int xyz = 0 neither alters how a compiler analyzes this variable nor changes any computations or states in the program ; it only hampers our understanding of this variable ’ s role in the program . Modifying a very small number of such aspects of a program marginally affects how we comprehend it , thus providing a way to produce changes imperceivable to both humans and a compiler . In this work , we view adversarial perturbations to programs as a special case of applying obfuscation transformations to them . Having identified a set of candidate transformations which produce imperceivable changes , a specific subset needs to be chosen in a way which would make the transformed program adversarial . Recent attempts ( Yefet et al. , 2019 ; Ramakrishnan et al. , 2020 ; Bielik & Vechev , 2020 ) which came closest to addressing this problem did not offer any rigorous formulation . They recommended using a variety of transformations without presenting any principled approach to selecting an optimal subset of transformations . We present a formulation which when solved provides the exact location to transform as well as a transformation to apply at the location . Figure 1 illustrates this . A randomly selected local-variable ( name ) when replaced by the name virtualname , which is generated by the stateof-the-art attack generation algorithm for programs ( Ramakrishnan et al. , 2020 ) , is unable to fool a program summarizer ( which predicts set item ) unless our proposed site optimization is applied . We provide a detailed comparison in Section 2 . In our work , we make the following key contributions – • We identify two problems central to defining an adversarial program – identifying the sites in a program to apply perturbations on , and the specific perturbations to apply on the selected sites . These perturbations are involve replacing existing tokens or inserting new ones . • We provide a general mathematical formulation of a perturbed program that models site locations and the perturbation choice for each location . It is independent of programming languages and the task on which a model is trained , while seamlessly modeling the application of multiple transformations to the program . • We propose a set of first-order optimization algorithms to solve our proposed formulation efficiently , resulting in a differentiable generator for adversarial programs . We further propose a randomized smoothing algorithm to achieve improved optimization performance . • Our approach demonstrates a 1.5x increase in the attack success rate over the state-of-the-art attack generation algorithm ( Ramakrishnan et al. , 2020 ) on large datasets of Python and Java programs . • We further show that our formulation provides better robustness against adversarial attacks compared to the state-of-the-art when used in training an ML model . 2 RELATED WORK . Due to a large body of literature on adversarial attacks in general , we focus on related works in the domain of computer programs . Wang & Christodorescu ( 2019 ) , Quiring et al . ( 2019 ) , Rabin et al . ( 2020 ) , and Pierazzi et al . ( 2020 ) identify obfuscation transformations as potential adversarial examples . They do not , however , find an optimal set of transformations to deceive a downstream model . Liu et al . ( 2017 ) provide a stochastic optimization formulation to obfuscate programs optimally by maximizing its impact on an obscurity language model ( OLM ) . However , they do not address the problem of adversarial robustness of ML models of programs , and their formulation is only to find the right sequence of transformations which increases their OLM ’ s perplexity . They use an MCMC-based search to find the best sequence . Yefet et al . ( 2019 ) propose perturbing programs by replacing local variables , and inserting print statements with replaceable string arguments . They find optimal replacements using a first-order optimization method , similar to Balog et al . ( 2016 ) and HotFlip ( Ebrahimi et al. , 2017 ) . This is formed - two replace sites corresponding to local variables b and r , and three insert sites at locations I1 , I2 , I3 . Ω is a vocabulary of tokens which can be used for the transformations . ( c ) This is a perturbed program with the tokens world and set from Ω used to replace tokens b and at location I3 . These transformations do not change the original functionality of P , but cause an incorrect prediction delete ( d ) Examples of two site selection vectors zi , zii selecting different components . zi = 1 for a location i signifies that the ith token in P is selected to be optimally transformed . zi corresponds to the perturbed program in ( c ) . an improvement over Zhang et al . ( 2020 ) , who use the Metropolis-Hastings algorithm to find an optimal replacement for variable names . Bielik & Vechev ( 2020 ) propose a robust training strategy which trains a model to abstain from deciding when uncertain if an input program is adversarially perturbed . The transformation space they consider is small , which they search through greedily . Moreover , their solution is designed to reason over a limited context of the program ( predicting variable types ) , and is non-trivial to extend to applications such as program summarization ( explored in this work ) which requires reasoning over an entire program . Ramakrishnan et al . ( 2020 ) extend the work by Yefet et al . ( 2019 ) and is most relevant to what we propose in this work . They experiment with a larger set of transformations and propose a standard min-max formulation to adversarially train robust models . Their inner-maximizer , which generates adversarial programs , models multiple transformations applied to a program in contrast to Yefet et al . ( 2019 ) . However , they do not propose any principled way to solve the problem of choosing between multiple program transformations . They randomly select transformation operations to apply , and then randomly select locations in the program to apply those transformations on . We instead show that optimizing for locations alone improves the attack performance . Further , we propose a joint optimization problem of finding the optimal location and optimal transformation , only the latter of which Ramakrishnan et al . ( 2020 ) ( and Yefet et al . ( 2019 ) ) address in a principled manner . Although formally unpublished at the time of preparing this work , we compare our experiments to Ramakrishnan et al . ( 2020 ) , the state-of-the-art in evaluating and defending against adversarial attacks on models for programs , and contrast the advantages of our formulation . 3 PROGRAM OBFUSCATIONS AS ADVERSARIAL PERTURBATIONS . In this section , we formalize program obfuscation operations , and show how generating adversarial programs can be cast as a constrained combinatorial optimization problem . Program obfuscations . We view obfuscation transformations made to programs as adversarial perturbations which can affect a downstream ML/DL model like a malware classifier or a program summarizer . While a variety of such obfuscation transformations exist for programs in general ( see section 2A , Liu et al . ( 2017 ) ) , we consider two broad classes – replace and insert transformations . In replace transformations , existing program constructs are replaced with variants which decrease readability . For example , replacing a variable ’ s name , a function parameter ’ s name , or an object field ’ s name does not affect the semantics of the program in any way . These names in any program exclusively aid human comprehension , and thus serve as three replace transformations . In insert transformations , we insert new statements to the program which are unrelated to the code it is inserted around , thereby obfuscating its original intent . For example , including a print statement with an arbitrary string argument does not change the semantics of the program in any way . Our goal hence is to introduce a systematic way to transform a program with insert or replace transformations such that a trained model misclassifies a program P that it originally classified correctly . Site-selection and Site-perturbation – Towards defining adversarial programs . Before we formally define an adversarial program , we highlight the key factors which need to be considered in our formulation through the example program introduced in Figure 2 . Consider applying the following two obfuscation transformations on the example program P in Figure 2.a – replacing local variable names ( a replace transform ) , and inserting print statements ( an insert transform ) . The two local variables b and r in P are potential candidates where the replace transform can be applied , while a print statement can potentially be inserted at the three locations I1 , I2 , I3 ( highlighted in Figure 2.b ) . We notate these choices in a program as sites– locations in a program where a unique transformation can be applied . Thus , in order to adversarially perturb P , we identify two important questions that need to be addressed . First , which sites in a program should be transformed ? Of the n sites in a program , if we are allowed to choose at most k sites , which set of ≤ k sites would have the highest impact on the downstream model ’ s performance ? We identify this as the site-selection problem , where the constraint k is the perturbation strength of an attacker . Second , what tokens should be inserted/replaced at the k selected sites ? Once we pick k sites , we still have to determine the best choice of tokens to replace/insert at those sites which would have the highest impact on the downstream model . We refer to this as the site-perturbation problem . Mathematical formulation . In what follows , we propose a general and rigorous formulation of adversarial programs . LetP denote a benign program which consists of a series of n tokens { Pi } ni=1 in the source code domain . For example , the program in Figure 2.a , when read from top to bottom and left to right , forms a series of n = 12 tokens { def , b , . . . , r , + , 5 } . We ignore white spaces and other delimiters when tokenizing . Each Pi ∈ { 0 , 1 } |Ω| here is considered a one-hot vector of length |Ω| , where Ω is a vocabulary of tokens . Let P ′ define a perturbed program ( with respect to P ) created by solving the site-selection and site-perturbation problems , which use the vocabulary Ω to find an optimal replacement . Since our formulation is agnostic to the type of transformation , perturbation in the remainder of this section refers to both replace and insert transforms . In our work , we use a shared vocabulary Ω to select transforms from both these classes . In practice , we can also assign a unique vocabulary to each transformation we define . To formalize the site-selection problem , we introduce a vector of boolean variables z ∈ { 0 , 1 } n to indicate whether or not a site is selected for perturbation . If zi = 1 then the ith site ( namely , Pi ) is perturbed . If there exist multiple occurrences of a token in the program , then all such sites are marked 1 . For example , in Figure 2.d , if the site corresponding to local variable b is selected , then both indices of its occurrences , z3 , z9 are marked as 1 as shown in zi . Moreover , the number of perturbed sites , namely , 1T z ≤ k provides a means of measuring perturbation strength . For example , k = 1 is the minimum perturbation possible , where only one site is allowed to be perturbed . To define site-perturbation , we introduce a one-hot vector ui ∈ { 0 , 1 } |Ω| to encode the selection of a token from Ω which would serve as the insert/replace token for a chosen transformation at a chosen site . If the jth entry [ ui ] j = 1 and zi = 1 , then the jth token in Ω is used as the obfuscation transformation applied at the site i ( namely , to perturb Pi ) . We also have the constraint 1Tui = 1 , implying that only one perturbation is performed at Pi . Let vector u ∈ { 0 , 1 } n×|Ω| denote n different ui vectors , one for each token i in P . Using the above formulations for site-selection , site-perturbation and perturbation strength , the perturbed program P ′ can then be defined as P ′ = ( 1− z ) · P + z · u , where 1T z ≤ k , z ∈ { 0 , 1 } n , 1Tui = 1 , ui ∈ { 0 , 1 } |Ω| , ∀i , ( 1 ) where · denotes the element-column wise product . The adversarial effect of P ′ is then measured by passing it as input to a downstream ML/DL model θ and seeing if it successfully manages to fool it . Generating a successful adversarial program is then formulated as the optimization problem , minimize z , u ` attack ( ( 1− z ) · P + z · u ; P , θ ) subject to constraints in ( 1 ) , ( 2 ) where ` attack denotes an attack loss . In this work , we specify ` attack as the cross-entropy loss on the predicted output evaluated at P ′ in an untargeted setting ( namely , without specifying the prediction label targeted by an adversary ) ( Ramakrishnan et al. , 2020 ) . One can also consider other specifications of ` attack , e.g. , C & W untargeted and targeted attack losses ( Carlini & Wagner , 2017 ) .

Summary: This work tackles the problem of adversarial attacks against ML models for code-understanding tasks, such as function summarization. It formulates the problem as the adversarial application of existing semantics-preserving program transformations (e.g., renaming variables), by jointly optimizing on the location of such a transformation, and the argument to the transformation (e.g., what to replace an existing identifier with). It shows that such adversarial examples increase the attack success rate over baseline approaches, and training with such examples increases the robustness of the resulting model to the same or baseline attacks.