STRATA: Simple, Gradient-free Attacks for Models of Code

1 INTRODUCTION . Although machine learning has been shown to be effective at a wide variety of tasks across computing , statistical models are susceptible to adversarial examples . Adversarial examples , first identified in the continuous domain by Szegedy et al . ( 2014 ) , are imperceptible perturbations to input that result in misclassification . Researchers have developed effective techniques for adversarial example generation in the image domain ( Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2017 ; Papernot et al. , 2016a ) and in the natural language domain ( Alzantot et al. , 2018 ; Belinkov & Bisk , 2018 ; Cheng et al. , 2020 ; Ebrahimi et al. , 2018 ; Michel et al. , 2019 ; Papernot et al. , 2016b ) , although work in the source code domain is less extensive ( see Related Work ) . The development of adversarial examples for deep learning models has progressed in tandem with the development of methods to make models which are robust to such adversarial attacks , though much is still being learned about model robustness ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ; Shafahi et al. , 2019 ; Wong et al. , 2019 ) . The threat of adversarial examples poses severe risks for ML-based malware defenses ( Al-Dujaili et al. , 2018 ; Grosse et al. , 2016 ; Kaur & Kaur , 2015 ; Kolosnjaji et al. , 2018 ; Kreuk et al. , 2019 ; Suciu et al. , 2019 ) , and introduces the ability of malicious actors to trick ML-based code-suggestion tools to suggest bugs to an unknowing developer ( Schuster et al. , 2020 ) . Thus , developing state-of-the-art attacks and constructing machine learning models that are robust to these attacks is important for computer security applications . Generating adversarial examples for models of code poses a challenge compared to the image and natural language domain , since the input data is discrete and textual and adversarial perturbations must abide by strict syntactical rules and semantic requirements . The CODE2SEQ model is a state-of-the-art model of code that has been used to explore adversarial example design and robustness methods on models of code ( Rabin & Alipour , 2020 ; Ramakrishnan et al. , 2020 ) . In this work , we propose the Simple TRAined Token Attack ( STRATA ) , a novel and effective method for generating black-box and white-box adversarial attacks against CODE2SEQ . Our method replaces local variable names with high impact candidates that are identified by dataset statistics . It can also be used effectively for targeted attacks , where the perturbation targets a specific ( altered ) output classification . Further , we demonstrate that adversarial training , that is , injecting adversarial examples into CODE2SEQ ’ s training set , improves the robustness of CODE2SEQ to adversarial attacks . We evaluate STRATA on CODE2SEQ , though we hypothesize that the method can be applied to other models . The principles underlying STRATA apply not only to models of source code , but also to natural language models in contexts where the vocabulary is large and there is limited training data . STRATA has a number of advantages compared to previously proposed adversarial attack strategies : 1 . STRATA constructs state-of-the-art adversarial examples using a gradient-free approach that outperforms gradient-based methods ; 2 . STRATA generates white-box adversarial examples that are extremely effective ; blackbox attacks that use dictionaries created from unrelated code datasets perform similarly ( Appendix C ) 3 . STRATA does not require the use of a GPU and can be executed more quickly than competing gradient-based attacks ( Appendix D.1 ) ; 4 . STRATA is the only available method ( known to the authors at present ) which performs targeted attacks on CODE2SEQ , which is the current state-of-the-art for models of code . 2 MOTIVATION . CODE2SEQ , developed by Alon et al . ( 2019a ) , is an encoder-decoder model inspired by SEQ2SEQ ( Sutskever et al. , 2014 ) ; it operates on code rather than natural language . CODE2SEQ is the state-ofthe-art code model , and therefore it represents a good target for adversarial attacks and adversarial training . The model is tasked to predict method names from the source code body of a method . The model considers both the structure of an input program ’ s Abstract Syntax Trees ( ASTs ) as well as the tokens corresponding to identifiers such as variable names , types , and invoked method names . To reduce the vocabulary size , identifier tokens are split into subtokens by commonly used delimiters such as camelCase and under_scores . In this example , subtokens would include “ camel ” and “ case ” and “ under ” and “ scores ” . CODE2SEQ encodes subtokens into distributed embedding vectors . These subtoken embedding vectors are trained to capture semantic structure , so nearby embedding vectors should correspond to semantically similar subtokens ( Bengio et al. , 2003 ) . In this paper , we distinguish between subtoken embedding vectors and token embedding vectors . Subtoken embedding vectors are trained model parameters . Token embedding vectors are computed as a sum of the embedding vectors of the constituent subtokens . If the token contains more than five subtokens , only the first five are summed , as per the CODE2SEQ architecture . The full description and architecture of the CODE2SEQ model is given in the original paper by Alon et al . ( 2019a ) . The CODE2SEQ model only updates a subtoken embedding as frequently as that subtoken appears during training , which is proportional to its representation in the training dataset . However , the training datasets have very large vocabularies consisting not only of standard programming language keywords , but also a huge quantity of neologisms . The frequency at which subtokens appear in the CODE2SEQ java-large training set varies over many orders of magnitude , with the least common subtokens appearing fewer than 150 times , and the most common over 108 times . Thus , subtoken embedding vectors corresponding with infrequently-appearing subtokens will be modified by the training procedure much less often than common subtokens . Figure 1a demonstrates this phenomenon , showing a disparity between L2 norms of frequent and infrequently-appearing subtoken embedding vectors . We confirm this empirically . When we initialized embedding vectors uniformly at random and then trained the model as normal , as per Alon et al . ( 2019a ) , we found that the vast majority of final , i.e. , post-training , embedding vectors change very little from their initialization value . In fact , 90 % of embedding tokens had an L2 distance of less than 0.05 between the initial vector and final , post-training vector when trained on a java dataset . About 10 % of subtokens had a large L2 distance between the initial embedding and final embedding ; these subtokens were more frequent in the training dataset and had embedding vectors with a notably larger final L2 magnitude ( Figure 1 ) . The observation that high-L2-norm embedding vectors are associated with subtokens that appear sufficiently frequently in the dataset motivates the core intuitions of our attack1 . We show in this paper that subtokens with high-L2-norm embedding vectors can be used for effective adversarial examples , which are constructed as follows : 1We note very-high-frequency subtokens have small L2 norms . Examples of these very-high-frequency subtokens include : get , set , string , and void , which appear so often as to not be useful for classification . Despite the fact that these subtokens are not good adversarial candidates for STRATA , there are so few of them that we expect them to have minimal influence on the effectiveness of our attack . 1 . To maximize adversarial effectiveness in a white-box setting , we should use tokens with high L2 norm embedding vectors as local variable name replacements . We confirm this empirically in the Experiments section . 2 . In the absence of information about the L2 norms of embedding vectors , we can isolate high-L2-norm subtokens for local variable name replacement by selecting tokens which appear in the training dataset often enough to be well trained . This is empirically confirmed by the large intersection of high-L2-norm subtokens and subtokens with high frequency . 3 METHODS . 3.1 DATASET DEFINITIONS AND CONSIDERATIONS . We evaluate our attack on four datasets that are used for training different CODE2SEQ models . There are three , non-overlapping Java datasets : java-small ( 700k examples ) , java-medium ( 4M examples ) , and java-large ( 16M examples ) ( Alon et al. , 2019a ) , and one Python dataset , python150k ( Raychev et al. , 2016 ) . We disambiguate the trained CODE2SEQ models for each datasets by denoting them CODE2SEQ-SM , CODE2SEQ-MD , CODE2SEQ-LG , and CODE2SEQ-PY for models trained on java-small , -medium , -large , and python150k respectively . Many of our experiments are evaluated on all four models ; however , experiments that require adversarial training are only evaluated on CODE2SEQ-SM , for computational feasibility . 3.2 THE ATTACK . Traditional adversarial attacks on discrete spaces involve searching the discrete space for semantically similar perturbations that yield a misclassification . Searching the space of all possible valid discrete changes in source code is often intractable or even impossible ( Rice , 1953 ) . However , there are strategies to reduce the search space . For example , perturbations may be limited to a small number of predefined operations which are known to be semantically equivalent , such as inserting deadcode , replacing local variable names , or replacing expressions with known equivalent expressions . Gradientbased attacks on the embedding space may also be used in order to optimize the search of the space itself ( Ramakrishnan et al. , 2020 ; Yefet et al. , 2020 ) . However , gradient-based attacks are computationally expensive and rely heavily on knowledge of the exact parameters of the model . We propose STRATA , which replaces local variable names with high-impact subtokens to generate adversarial examples . STRATA leverages the fact that only a relatively small number of subtoken embeddings are critical for the classification task performed by the CODE2SEQ model . In Section 2 , we presented two ways to identify high-impact subtokens . STRATA will use these to replace local variable names . Recall that the model composes these subtokens into tokens by summing the first five constituent subtoken embedding vectors . We wish to maximize the L2 norm of the resulting token , while minimizing the semantic change . We propose three strategies : 1. single : pick a single subtoken as the token ; 2 . 5-diff : pick five different ( not necessarily unique ) subtokens and concatenate them , which will have a higher expected L2 norm than single ; 3 . 5-same : pick a single subtoken , and repeat the subtoken five times to form a token , which will have the largest expected L2 norm , by the triangle inequality2 . We subjectively propose that single is the smallest and most realistic semantic change , 5-same is the largest change and the “ best-case '' for an adversarial example , and 5-diff represents an intermediate attack strength . For a given method , STRATA generates an untargeted perturbation as follows : 1 . Select one random local variable v ; 2 . Choose an adversarial token v∗ appropriately , using the chosen concatenation strategy ( single , 5-diff , or 5-same ) . For white-box attacks , choose each subtoken from a high-L2norm vocabulary ( top-n by L2 norm ) . For black-box attacks , choose each subtoken with sufficiently high frequency ( top-n by frequency ) . We discuss the optimal cutoff values ( n ) for L2 and frequency in Section 3.4 . 3 . Replace v with v∗ . For attacks on the Python dataset , since determining whether a variable is local or non-local is not always possible by looking at only the body of the method , we treat all variables as local . To perform targeted attacks in which we want the output to include a particular subtoken t , we perform the same steps as the untargeted attack , and choose v∗ to be a 5-same concatenation of t .

Summary: This paper proposes STRATA, a simple adversarial attack against the code2seq model. The key idea is to replace local variable names in the input code with other randomly chosen sub-tokens with embedding vectors of relatively high L2 norms. Meanwhile, they observe that such tokens often appear frequently in the training set, thus alternatively they can simply use frequently appeared tokens as the target to perform the attacks. In this way, they can attack the model in the black-box scenario, without the knowledge of the model parameters and the training data, as long as they can roughly approximate the frequency distribution of different code sub-tokens in the training set. They evaluate their approach on code2seq models trained for the Java code, and compare with existing attacks against code2seq models. They first show that the 5-same attack, i.e., repeating a sub-token 5 times and concatenating them as the new local variable name, is the most effective attack. This attack decreases the F1 scores more compared to the baseline attack from prior work. In addition, they show that by adding STRATA adversarial examples for adversarial training, the new model becomes more robust to their proposed attacks.