STRATA: Simple, Gradient-free Attacks for Models of Code
Authors: Jacob M. Springer
Bryn Marie Reinstadler
Una-May O'Reilly
Venue:  (2021)
Keywords: Deep Learning
Models of Code
Black-box Adversarial Attacks
Adversarial Robustness

Abstract: Adversarial examples are imperceptible perturbations in the input to a neural model that result in misclassification. Generating adversarial examples for source code poses an additional challenge compared to the domains of images and natural language, because source code perturbations must adhere to strict semantic guidelines so the resulting programs retain the functional meaning of the code. We propose a simple and efficient gradient-free method for generating state-of-the-art adversarial examples on models of code that can be applied in a white-box or black-box setting. Our method generates untargeted and targeted attacks, and empirically outperforms competing gradient-based methods with less information and less computational effort.

Under review as a conference paper at ICLR 2021
STRATA: SIMPLE, GRADIENT-FREE ATTACKS FOR MODELS OF CODE
Anonymous authors Paper under double-bling review
ABSTRACT
Adversarial examples are imperceptible perturbations in the input neural model that result in misclassification. Generating adversarial examples for source code poses additional challenge compared the domains of images and natural lan- guage, because source code perturbations must adhere to strict semantic guidelines so the resulting programs retain the functional meaning of the code. We propose a simple and efficient gradient- free method for generating satte-of-the-aan adversarial examples on models of code that can applied in white-box or black-box setting. Our method generates untargeted and targeted attacks, and empirically outperforms competing gradient-baa based methods with less information and less computational effort.
INTRODUCTION
Although machine learning has been shown to effective at a wide variety of tasks across computing, statistical models susceptible adversarial examples. Adversarial examples, first identified in the continuous domain by [Szeedyy et al. (2014), are imperceptible perturbations to input that
result in misclassification Researchers have developed effective techniques for adversarial example generation in the image domain (Goodfellow et al. 2015; Moosavi-Dezoooll et al.) [2017] Papernol 60) and in the natural language domain (Alzantot al. 2018]] Belinkov & Bisk 2018
al.
Cheng et al. 2020 Ebrahimi et al.) 2018 Michel et al. 2019 Papernot et al. 2016b), although work
the source code domain less extensive (see Related Work). The development of adversarial examples for deep learning models has progressed in tandem with the development of methods to make models which are robust to such adversarial attacks, though much still being learned about
model robustness (Goodfellow et al. 2015 Madry al. 2018 Shafahi et al. 2019 Wong et al. 2019).
The threat of adversarial examples poses severe risks for ML-based malware defenses (Al-Dujal)
et al. 2018 Grosse et al. 2016 Kaur & Kaur 2015 Kolosnjaji et al., 2018 Kreuk et al., 2019) Suciu al. 2019 and introduces the ability of malicious actors trick ML-based code- suggestion tools to suggest bugs to an unknowing developer (Schuster et al., 2020). Thus, developing state-of-the-art attacks and constructing machine learning models that are robust to these attacks is important for computer security applications. Generating adversarial examples for models of code poses a challenge compared the image and natural language domain, since the input data discrete and textual and adversarial perturbations must abide by strict syntactical rules and semantic requirements. The CODE2SEQ model is ofate-of-the-art model of code that has been used to explore adversarial example design and robustness methods on models of code (Rabin & Alipour 2020 Ramakrishnan et al.) 2020).
In this work, we propose the Simple TRAined Token Attack (STRATA), a novel and effective method for generating black-box and white-box adversarial attacks against CODE2sFQ. Our method replaces local variable names with high impact candidates that identified by dataset statistics. It can also be used effectively for targeted attacks, where the perturbation targets a specific (altered) output classification. Further, we demonstrate that adversarial training, that is, injecting adversarial examples into CODE2SEQ's 's training set, improves the robustness of CODE2SEQ to adversarial attacks.
We evaluate STRATA CODE2SEO, though hypothesize that the method can be applied to other models. The principles underlying STRATA apply not only to models of source code, but also to natural language models in contexts where the vocabulary is large and there is limited training data.

---

Under review as a conference paper at ICLR 2021
STRATA has number of advantages compared to previously proposed adversarial attack strategies:
1. STRATA constructs state-of-the-an art adversarial examples using a gradient-free approach that outperforms gradient-based methods;
2. STRATA generates white-box adversarial examples that are extremely effective; black- box attacks that use dictionaries created from unrelated code datasets perform similarly (Appendix C)
3. STRATA does not require the use of a GPU and can be executed more quickly than competing gradient- based attacks (Appendix D.I):
4. STRATA is the only available method (known to the authors at present) which performs
targeted attacks on CODE2SEQ, which is the current state-of-the-an for models of code.
MOTIVATION
CODE2SEQ. developed by Alon et al. (2019a), is an encoder-decoder model inspired by SEQ2SEQ (Sutskever et al. 2014); it operates on code rather than natural language, CODE2SEQ is the state-of- the-art code model, and therefore it represents a good target for adversarial attacks and adversarial training. The model is tasked to predict method names from the source code body of a method. The model considers both the structure of input program's 's Abstract Syntax Trees (ASTs) as well as the tokens corresponding to identifiers such as variable names, types, and invoked method names. To reduce the vocabulary size, identifier tokens are split into subtokens by commonly used delimiters such as camelCase and under under_scores. scores. In this example, subtokens would include "camel" and "case" and "under" and "scores". CODE2SEQ encodes subtokens into distributed embedding vectors. These subtoken embedding vectors are trained to capture semantic structure, so nearby embedding vectors should correspond semantically similar subtokens (Bengio al, 2003). In this paper, we distinguish between subtoken embedding vectors and token embedding vectors. Subtoken embedding vectors are trained model parameters. Token embedding vectors are computed as a sum of the embedding vectors of the constituent subtokens. If the token contains more than five subtokens, only the first five are summed, as per the CODE2SEQ architecture. The full description and architecture of the CODE2SEQ model is given in the original paper by Allnn et al. (2019a).
The CODE2SEQ model only updates a subtoken embedding as frequently as that subtoken appears during training, which is proportional to its representation in the training dataset. However, the training datasets have very large vocabularies consisting not only of standard programming language keywords, but also huge quantity of neologisms. The frequency at which subtokens appear in the cOdE2sEQ java-large training set varies over many orders of magnitude, with the least common subtokens appearing fewer than 150 times, and the most common over 10 times.
Thus, subtoken embedding vectors corresponding with enfrequeentyyaappeaiee subtokens will be modified by the training procedure much less often than common subtokens. Figure La demonstrates this phenomenon, showing a disparity between L2 norms of frequent and prreequently-apperrin subtoken embedding vectors.
We confirm this empirically. When we initialized embedding vectors uniformly at random and then trained the model as normal, as per Alon et al.] (2019a), we found that the vast majority of final, i.e., post-training, embedding vectors change very little from their initialization value. In fact, 90% of embedding tokens had an L.2 distance of less than 0.05 between the initial vector and final, post-training vector when trained on java dataset. About 10% of subtokens had a large L2 distance between the initial embedding and final embedding; these subtokens were more frequent in the training dataset and had embedding vectors with notably larger final L2 magnitude (Figure IT.
The observation that high-L2-norm embedding vectors are associated with subtokens that appear sufficiently frequently in the dataset motivates the core intuitions of our attack" We show in this paper that subtokens with high-L2-norm embedding vectors can be used for effective adversarial examples, which are constructed follows:
We note very- -igh- feggh-feeueecy.. subtokens have small L2 norms. Examples of these very- high-frequency subtokens include: get. set, string. and void. which appear so often as to not be useful for classification. Despite the fact that these subtokens are not gond adversarial candidates for STRATA, there are so few of them that we expect them have minimal influence the effectiveness of our attack.
2

---

Under review as a conference paper at ICLR 2021
L2 nom
90th percertio
Log frequency
Log frequency
(a) The 1.2 distance between pre- and post-training embedding vectors for each subtoken, plotted against the log frequency of subtokens in the java-small train- ing dataset.
(b) L.2 norm of trained subtoken embedding vectors. Dashed and dotted lines chosen arbitrarily to illustrate tokens which have high L2 norm frequency.
Figure 1: Visualization of embedding vectors for CODE2SEQ-SM.
To maximize adversarial effectiveness in a white-box setting, we should use tokens with high L2 norm embedding vectors local variable name replacements. We confirm this empirically in the Experiments section.
2. In the absence of information about the L2 norms of embedding vectors, high-L2-norm subtokens for local variable name replacement by selecting tokens which appear in the training dataset often enough to be well trained. This is empirically confirmed by the large intersection of high-L2-norm subtokens and subtokens with high frequency.
isolate
3. METHODS
3.1 DATASET DEFINITIONS AND CONSIDERATIONS
We evaluate our attack on four datasets that are used for training different CODE2SEQ models. There are three, non-overlapping Java datasets: java-small (700k examples), java-medium javv (4M examples). and java-large (16M examples) (Alon et al. 2019a). and one Python dataset, python (Raychev et al. 2016). We disambiguate the trained CODE2SEQ models for each datasets by denoting them CODE2SEQ SM, CODE2SEQ-ME MD, CODB2SEQ-LG, and CODE2SEQ-P for models trained on java-small, -medium, llarge, and python 1 50k respectively. Many of our experiments are evaluated on all four models; however, experiments that require adversarial training are only evaluated on CODE2SEQ-SA for computational feasibility.
3.2 THE ATTACK
Traditional adversarial attacks on discrete spaces involve searching the discrete space for semantically similar perturbations that yield misclassification. Searching the space of all possible valid discrete changes in source code is often intractable or even impossible (Rice) 1953). However, there are strategies reduce the search space. For example, perturbations may be limited predefined operations which are known to be semantically equivalent, such as inserting deadcode, replacing local variable names, or replacing expressions with known equivalent expressions. Gradient- based attacks on the embedding space may also be used in order to optimize the search of the space itself (Ramakrishnan et al., 2020 Yefet et al.) 2020). However, gradient-based attacks are computationally expensive and rely heavily knowledge of the exact parameters of the model.
small number of
We propose STRATA, which replaces local variable names with high-impen inphiimpact subtokens to generate adversarial examples. STRATA leverages the fact that only relatively small number of subtoken embeddings are critical for the classification task performed by the CODE2SEQ model.
In Section 2 presented two ways to identify high-impact subtokens. STRATA will these to replace local variable names. Recall that the model composes these subtokens into tokens by summing the first five constituent subtoken embedding vectors. We wish to maximize the L2 norm of the resulting token, while minimizing the semantic change. We propose three strategies:
1. single: pick single subtoken the token;
3

---

Under review as a conference paper at ICLR 2021
2. 5-diff: pick five different (not necessarily unique) subtokens and concatenate them, which will have a higher expected L2 norm than single;
3. 5-same: pick single subtoken, and repeat the subtoken five times to form a token, which will have the largest expected 1.2 norm, by the triangle inequality
We subjectively propose that single the smallest and most realistic semantic change, 5-same the largest change and the "best-case" for an adversarial example, and 5-diff represents an intermediate attack strength.
For given method, STRATA generates an untargeted perturbation as follows:
1. Select one random local variable e:
2. Choose an adversarial token v* appropriately, using the chosen concatenation strategy (single, 5-diff, or 5-same). For white-box attacks, choose each subtoken from high-L.2- norm vocabulary (top-n by L2 norm). For black-box attacks, choose each subtoken with sufficiently high frequency (top-n by frequency). We discuss the optimal cutoff values (n)
for L2 and frequency in Section 3.4
3. Replace U with v"
For attacks the Python dataset, since determining whether variable is local or non-local is not always possible by looking at only the body of the method, we treat all variables as local.
To perform targeted attacks in which want the output to include particular subtoken t, perform the same steps as the untargeted attack, and choose to be a 5-same concatenation t.
3.3 DATASET-AGNOSTIC BLACK-BOX ATTACKS
STRATA can generate effective adversarial attacks even without the training dataset. We can determine subtoken frequency statisties from a different (potentially non-overlapping) dataset. We empirically confirm that STRATA can use non-overlapping datasets (java-small, java-medium, and java-large) in this way attack CODE2SEQ models that have been trained different Java dataset. We conclude that the subtoken distributions of non-overlapping Java datasets are sufficiently similar for STRATA be effective (Appendix C). This confirms that STRATA can be applied in a completely black-box setting without knowledge of the model parameters or training dataset.
3.4 IMPLEMENTATION
We assayed the effectiveness of the method by applying a simple local variable token replacement attack. We select single locally-declared variable random and rename it with adversarial token which does not conflict with another name. Because the initial variable is locally declared, we know that changing the effect. The token replacement, however, can effectively attack CODE2SEO; example shown in the Appendix.
will have effect elsewhere in the program, and will have behavioral
The java-small test set consists of -57,000 examples; we exclude from our testing dataset all methods that cannot be attacked by our method, i.e. all methods without local variables, leaving ~40,000 examples. The python 150k dataset consists of 50,000 testing examples. We then create several vocabularies of subtokens from which choose adversarial substitutions:
1. All: Contains all subtokens. Note that the number of subtokens varies by dataset (Table I:
2. Top n by L2 norm contains subtokens for which their L2 norm embedding vectors are the n highest;
3. Top n by frequency contains only the n subtokens which occur in the training data with highest frequency.
To obtain optimal thresholds of n, we swept the range of possibilities to find the n that minimizes F1 score, i.e., generates the best performing adversarial examples (Figure We present the final values of n in [
The triangle inequality states that ||x + yll S ||x|| + llyl], and is equal (and thus maximized) when and y colinear, which occurs when y. This easily generalized to five vectors.
4

---

Under review as a conference paper at ICLR 2021
Dataset
Model name
Optimal n by frequency
Optimal n by
Total subtokens
javalsmall
CODE2SEQ-SM
1800 (2.51%)
1000 (1.39%)
71766
java-med
CODE2SEQ-MD
3000 (1.65 (1.65%)
3000 (1.65%)
181125
java-large
CODE2SEQ-LG
10000 (5.54%)
6000 (3.33%)
180355
python 150k
CODE2SEQ-LG
1600 (1.99%)
1600 (1.99%)
80336
Table 1: The name of the CODE2SEQ model trained on each dataset, along with optimal values of n for the top-n by frequency and L2 norm vocabularies, and the total number subtokens for each dataset.
0.48
0.46
F1 score
0.29
0.26
Random rank order Frequency rank order
0.38
Random rank order
Frequency rank order rank order
0.35
L2 rank order
1000o
20000
30000
40000
50020
0000
25000
500C0
190000
129000
150000 175000
Top-n parameter
Top-n parameter
(a) CODE2SEQ-SM
(b) CODE2SEQ-M
0.50
0.40
0.19
0.42
Random rank order Frequency rank order
Random rank order
0.30
Frequency order
L2 rank order
034
L2 rank order
0.36
20020
40020
e0deo
10030
100000 122000 140200 160000
Top-n parameter
Top-n parameter
(c) CODE2SEQ-1G
(d) CODE2SHO-PY
Figure 2: The F1 scores of the CODE2SEQ models, evaluated the java-small python testing dataset that has been adversarially perturbed with STRATA for given top-n parameters, using 5-same concatenation. A lower Fl score corresponds more effective attack.
4 EXPERIMENTS AND DISCUSSION
4.1 UNTARGETED ATTACKS
compares the three proposed concatenation strategies on each of the four proposed vocabular- ies (three in Java, one in Python). To perform comparisons, we measure the Fl score of the model on the java-small testing dataset, with each adversarial perturbation. Lower F1 scores correspond with better attacks. We see that the performance of CODE2sEQ drops when a local variable replaced with a token composed of random selection from all subtokens, using the 5-diff and 5-same concatenation strategies. However, we observe larger drop in Fl score when we make variable replacements from subtokens selected from the top-n subtokens by L2 norm or by frequency, using values of specified in Table [ When we replaced local variables with random tokens with high L2 norm, the Fl score dropped substantially, confirming our hypothesis that adversarial examples by selecting replacement tokens such that their subtoken embedding vectors have high 1.2 norm. Similarly, adversarial examples that replace local variable with subtokens of
improve the effectiveness of

---

Under review as a conference paper at ICLR 2021
high frequency in the training dataset highly effective, suggesting that the black-box method of choosing adversarial subtokens based on frequency alone can approximate the white-box attack.
Surprisingly, the Fl score of CODE2SEQ sM increased for random and frequency- based adversarial perturbations constructed with the single concatenation strategy, suggesting that CODE2SEQ-S relies less variable names for classification than CODE2SEQ-MD CODE2SEQ-LG. The attacks on CODE2SEQ-PY were also incredibly effective, although the baseline accuracy for that model was already lower than the rest of the models.
Baseline
All
Top-n by L2 norm
Top-n by Frequency
single
5-diff
5-same single
5-diff
5-same
single
5-diff
5-same
CODE2SEQ-SM
.369
.381
.350
.310
.362
.66
.214
.372
.284
.231
CODE2SEQ-MD
.564
.488
.531
.492
.513
.416
.375
.513
.385
.345
CODE2SEQ-LG
.608
.536
.547
.488
.542
.396
.360
.548
.427
.388
CODE2SEQ-PY .313
.536
.547
.488
.249
.198
.153
.256
.211
.172
Table 2: Evaluation of F1 scores of STRATA on our CODE2SEQ models, using vocabularies and replacement strategies described in the Methods section. The baseline refers the performance of the model on the original test dataset. For the Java models, samples were drawn from the java-small testing set such that each method included a local variable to perturb. All top-n scores use optimal values of proposed in Table [
As proposed Section 3.2 the effectiveness of the adversarial attack is optimized when we replace local variables with token that is constructed with the 5-same strategy. Strikingly, the black-box attack (top-n by frequency) is nearly as effective as the white-box attack (top-n by I.2).
4.2 TARGETED ATTACKS
We perform targeted adversarial attacks on the Java datasets that aim to inject a chosen subtoken into the output by performing a STRATA attack using the targeted subtoken for replacement. See Appendix D for an example.
To assay the effectiveness of targeted attacks, we perform targeted attacks that target three different subtoken vocabularies: (1) all valid subtokens, (2) the optimized 1.2 vocabulary, and (3) the opti- mized frequency vocabulary, where vocabularies are optimized for CODE2SEQ-SM. CODE2SEQ-MIT CODE2SEQ-LA appropriately. We determine that particular attack is successful if the selected subtoken is included the output. We measure the percent of successful attacks, thus computing an aggregate effectiveness of targeted attacks (Table 3) We omit targeted attacks the Python dataset although we expect that a similar trend holds.
Model
Perturbation
Strategy
% success
CODE2SEQ-LG
L2, top 6k Frequency, top 10k All
5-same
37.1
CODE2SEQ-LG
5-same
35.6
CODE2SEQ-LG
5-same
3.9
CODE2SEQ-MD
L2, top 3k Frequency, top 3k All
5-same
43.8
CODE2SEQ-MD
5-same
39.1
CODE2SE0-MD
5-same
1.4
CODE2SEQ-SM
L2, top lk
5-same
58.7
CODE2SEQ-SM Frequency, top 1.8k
5 5-same
52.8
CODE2SEQ-SM
All
5-same
2.1
Table 3: Effectiveness of targeted attacks CODE2SEQ.
Table Freveals that CODE2SEQ is especially vulnerable to targeted attacks performed on high-impact subtokens. The black-box (frequency) attack performs similarly to the white-box (I.2) attack.

---

Under review as a conference paper at ICLR 2021
4.3 TRANSFERABILITY
In this section we show that gradient-basee adversarial training as proposed by Ramakrishnan et al. (2020) not effective improving robustness STRATA attacks. We CODE2SEQ model that has been trained to be robust to the gradient- based adversarial examples proposed by Ramakrishnan et al. (2020), and find that the model is indeed robust the gradient- based token-reg leken-replacement perturbations. However, neither the original nor the robust model are impervious to perturbations produced by STRATA (Table J This result confirms that STRATA can effectively target models that are robust to some gradient- based perturbations; therefore it is useful tool when hardening models of code, even when gradient- -based perturbations are also being used.
Model No perturbations (FI) Gradient perturbations (FI)
STRATA perturbations (FI)
Original
.363
.243
.212
Robust .367
.342
.240
Table 4: F1 scores of a non-robust and robust CODE2SEQ-SM model on gradient- based adversarial perturbations and STRATA perturbations.
STRATA OUTPERFORMS SIMILAR ATTACKS
At the time of writing there are two other works that address adversarial attacks targeting CODE2SEQ: a class source code transformations by |Rabin & Alipour (2020) and gradient-based attacks by RRamakiishaan al. (2020). We greatly outperform the transformations considered in Rabin &
Alipour (2020) (Appendix B
We compare our work gradient-based adversarial perturbations proposed by Ramakkishhan et al. (202)) in which they attack a CODE2SEQ-SM model. We consider the variable replacement, print statement insertion. try-catch statement insertion, and worst-case single transformation attacks for our comparison. Note: to establish fair comparison, we include the 17,000 examples in the testing set that do not include local variable, and thus we do not even attempt to perturb them, hence why the F1 scores of the adversarial examples are larger than as reported in Table [ We find that STRATA outperforms all attacks performed by RRaaakishhann et al.) (2020) except for the worst-case transformation, which is inherently larger transformation than our varible-replacement attack. This includes greatly outperforming the gradient-based local variable replacement attack and performing similarly the worst-case transformation, despite the fact that STRATA generates smaller perturbations with less computational effort, and with less information about the model (Table These results indicate that STRATA attacks state-of-the-art on CODE2SEQ.
Adversarial attack
Fl
of baseline score
Baseline (Ramakrishnan et al.)
.444
100
Variable Replacement (Ramakrishnan al.)
.389
93,9
Try-catch Statement Insertion (Ramakrishnan et al.)
.336
81.2
Print Statement Insertion (Ramakrishnan al.)
.312
75.3
Worst-case Transformation (Ramakrishnan et al.)
.246
59.4
Baseline (STRATA)
.425
100
Top 1k by L2 (STRATA)
.316
74.4
Top 1.8k by frequency (STRATA)
.328
77.2
Table 5: Comparison F1 scores our method to gradient- based attacks described by Ramakrishnan al. (222)). Both our model and the Ramakrishnan et al. (2020) model are trained on java-small. We perform 5-same attacks.
7

---

Under review as a conference paper at ICLR 2021
6 RELATED WORK
veresaria examples for models of code of prior research on models of code. Several papers develop techniques for generating adversarial examples on models of source code: Quiring et al. (2019) perform adversarial attacks on source code by applying Monte-Carlo tree search over possible transformations, and Zhang et al. (2020) apply Eetropolis-Hassting
Allamanis et al. (2018) provide a comprehensive survey
sampling to perform identifier replacement to create adversarial examples. Bielik Vechev (2020) improve on the adversarial robustness of models of code by developing a model that can learn to abstain if uncertain.
Adversarial examples for CODE2VEC and CODE2SEQ Yefet et al. 2020) generate both gradient- based targeted and untargeted adversarial examples for CODE2VEC Alon al. (2019b). Most directly related to our paper, Ramakrishnan et al. (2020) perform gradient- gradient-based ad adversarial attacks and adversarial training on CODE2SEQ. Rabin & Alipour (2020) evaluate the robustness of CODE2SEQ to semantic-presenvve transformations.
Targeted attacks on models of code As previously noted, Yefet et al. (2020) propose a method for targeted adversarial examples for codE2vEc (Alon al. 2019b). To our knowledge, at the time of writing, no other paper performs targeted attacks on CODE2SEQ.
Adversarial training for robustness Many recent papers have examined the robustness of neural network models to adversarial perturbations. Szegedy et al. (2014) demonstrates that neural networks are vulnerable to adversarial perturbations, but that training on these perturbations, i.e., adversarial training, can increase robustness. Further papers explore faster and more effective methods of adversarial training to improve robustness, though mostly in the continuous domain (Madry et al. 2018. Shafahi et al. 2019 Wong et al.] 2019). RRamakrishaan et al.] (2020) perform adversarial training on CODE2SEQ to improve robustness. Yefet et al. (2020) propose multiple methods to improve robustness of models of source code, including adversarial training, outlier detection, and excluding variable names from model input.
7 CONCLUSION
In this work, we presented STRATA, simple, gradient-free method for generating adversarial examples discrete space that can be used to help build robustness models of code. Because the L2 norm of an embedding vector can be approximated by the frequency in the training data, STRATA can be used to generate gradicnt-fir white- and black-box attacks. We presented effective attacks using this method, including targeted attacks, and also showed that adversarial fine-tuning using STRATA examples can lead to increased robustness, even when the fine-tuned model is only being tested on clean data.
Our work does have some limitations. In the continuous domain, many adversarial attacks may not be detected by humans; in the discrete domain, this is not possible, but some attacks in the discrete space are more realistic (harder to spot) than others. The most powerful attack we propose, 5-same, is also the least realistic; single smaller perturbation but it has less effective attacks. We expect that the attack could be expanded by adding dummy local variables to methods that do initially contain a local variable, mitigating the current inability to attack methods without local variables.
STRATA does not just have application as an attack, but also as a defense. We present preliminary adversarial training results in the appendix which suggest that using adversarial examples generated with STR for adversarial training can improve robustness to adversarial attacks. Since we show that standard bbasedd adversarial training is ineffective at defending against STRATA, it is important for robust models to be adversarial training with STRATA adversarial examples to ensure comprehensive robustness. A complete study and evaluation of the effectiveness of adversarial training using STRATA an important future direction.
Another exciting of future inquiry would be the application of this attack to other models, including natural language models. The magnitude differences in subtoken frequency in code is similar to the magnitude differences in word frequency in natural language (Piantadosi) 2014). In fact,
8

---

Under review as a conference paper at ICLR 2021
tokens in software follow the same distribution as words in natural language (Zhang 2008 believe that our technique should be applicable in the NLP setting (with modification; for example, replacement dictionaries might only be filled with high- nigh-impact synonyms). We theorize that STRATA could be used to good effect in models of natural language as well as models of code, and that the basic intuitions underlying the attack can extend even to domains where the actual training data not available, as long as a suitably similar dataset is available. Our results stem from fundamental features of dataset distribution and training process, and they should generalize to other applications.
Thus
STRATA has a broad application to the field of machine learning in discrete spaces with large vocabularies. We present the insight that subtoken frequency correlates with L2 norm and thus can be used to determine high impact subtokens; this technique is not priori exclusive to code2seq or even models of code general since relies on properties of the dataset distribution and training technique and not the model architecture. STR is high-impact, low-cost attack strategy that can be used bolster robustness; we have made its code open-source so that those who wish to use may do so.
REFERENCES
Abdullah Al-Dujaili, Alex Huang, Erik Hemberg, and Mna-May O' O'Reilly. Adversarial Deep Learning for Robust Detection of Binary Encoded Malware. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 76-82, San Francisco. CA, May 2018. IEEE. ISBN 978-1-5386-8276-0, doi: 10.1109/ wwW.2018.00020 URL [RLts::s/eeeeeeoorr ieee. org wttp:///wwme/////2/22222
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4) 2018.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. In International Conference on Learning Representations, net http://oooonmmddllkeeeeeeee
2019a.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed rep- resentations of code. Proceedings of the ACM on Programming Languages, 3(POPL):1-29, 2019b.
Moustafa Alzantot, Yash Sharma Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine transla- tion. In International Conference on Learning Representations, 2018.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 137-1155, 2003.
Pavol Bielik and Martin Vechev. Adversarial robustness for code. arXiv preprint arXiv:2002.04694 2020.
Minhao Cheng, Jinfeng Yi, Pin- Yu Chen, Huan Zhang, and Cho-Jui Hsieh. Seq2Sick: Evaluating the Robustness of Sequence-to-Sen Se-quunne Models with Adversarial Examples. Proceedings of the AAAI Conference on Artificial Intelligence, (4((4))3601 April 2020. ISSN 2374-3468, 2159-5399. doi: 10. 1091/a0/a v34i04.5767. w3..0/////.........55777...
URL. / phpp /
AAAI CAAI/article/view//5/
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial exam- ples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 31-36, 2018.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations
(ICniionnnn(RRRR (ICLR), 2015.
Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick Mc-
Daniel. Adversarial Perturbations Against Deep Neural Networks for Malware Classification. arXiv: 04455 [cs], June 2016. URL http: //atx:///axiv.... www.arwwiw.oro.//aaaaoaeeeeeeaee 1606.04435.
org /1606 04435 arXiv:

---

Under review as a conference paper at ICLR 2021
Arvinder Kaur and Kamaldeep Kaur. An Empirical Study of Robustness and Stability of Ma- chine Learning Classifiers in Software Defect Prediction. In El-Sayed M. El-Alfy, Sabu M. Thampi, Hideyuki Takagi, Sclwyn Piramuthu, and Thomas Hanne (eds.), Advances in Intel- ligent Informatics, volume 320, pp. 383-397. Springer International Publishing, Cham, 2015. doi: 10. 10.100779978333399-2222 978 319 11218-22111 3 35 Series Title: Advances in Intelligent Systems and Computing.
35. URL hhttp://inkk spr nger. carer.cooo0o 1007/
Bojan Kolosnjaji, Ambra Demontis, Battista Biggio, Davide Maiorca, Giorgio Giacinto, Claudia Eckert, and Fabio Roli. Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables. In 2018 26th European Signal Processing Conference (EUSIPCO), pp. 533-537, Rome, September 2018. IEEE. ISBN 978-90-887970-11-5. doi: 10.23919/ESIICO. ieee htt..e/e.rroroooooeeee/////////e
2018.8553214.
Felix Kreuk, Assi Barak, Shir Aviv-Reuven, Moran Baruch, Benny Pinkas, and Joseph Keshet. Deceiv- ing End-to-End Deep Learning Malware Detectors using Adversarial Examples. [cs], January 2019, URL
arXiv: 1802.04528.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Paul Michel, Xian Li, Graham Neubig, and Juan Pino. On Evaluation of Adversarial Perturbations for Srrguence-to-Sqquenn Models. In Proceedings of the 2019 Conference of the North, pp. 3103-3114, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-133
Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal Adversarial Perturbations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 86-94, Honolulu, HI, July 2017. IEEE. ISBN 978- 1577----388--457-11 doi: 10. 109/ CVPR.2017.17. URL htlphttt::/// eeexp ore.
org http:///cccmeett/////e///
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (FuroS& ( P), pp. 372-387. IEEE, 2016a.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. Crafting adver- sarial input sequences for recurrent neural networks. MILCOM 2016-2016 IEEE Military Communications Conference, pp. 49-54. IEEE, 2016b.
Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112-11300 2014.
Erwin Quiring, Alwin Maier, and Konrad Rieck. Misleading authorship attribution of source code using adversarial learning. In 28th (USENIX) Security Symposium ((USENIX) Security 19), pp. 479-496, 2019.
Md. Rafiqul Islam Rabin and Mohammad Amin Alipour. Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preservinn Transformations. arXiv:2004.07315 [cs], April 2020.
2004 C7313
arXiv: 2004.07313.
Goutham Ramakrishnan, Jordan Henkel, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. Semantic Robustmess of Models of Source Code. arXiv:2002.0300 [cs, stat], June 2020.
URL
03043 arXiv: 2002.03043.
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, 51(10):331777, 2016.
Henry Gordon Rice. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2):358-366 1953.
Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. arXiv preprint arXiv:2007.02220 2020.
10

---

Under review as a conference paper at ICLR 2021
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pp. 3358-3369, ,,69, 2019.
Octavian Suciu, Scott E. Coull, and Jeffrey Johns. Exploring Adversarial Examples in Malware Detection. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 8-14, San Francisco, CA, USA, May 2019, IEEE. ISBN 978- 978-1-72813-508-3 doi: 10.1 10110/99..0099000000 URL htttps://ieeepllee ieeee.eeeee.....ee org/ /ttp///etttettt//s////////
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-31 1104-111,,, 12, 2014.
Christian Szegedy, Wojeiech Zaremba, Πya Sutskever, Joan Bruna Estrach, Dumitru Erhan, Ian Goodfellow, and Robert Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Eric Wong, Leslie Rice, and Zico Kolter. Fast better than free: Revisiting adversarial training. In International Conference Learning Representations, 2019.
Noam Yefet, Uri Alon, and Eran Yahav. Adversarial Examples for Models of Code. arXiv:1910.0751: 07517 arXiv: 1910.07517.
[cs], May 2020. URL httt:::/aaaiv..o//////aaaaa
Hongyu Zhang. Exploring regularity in source code: Software science and zipf"s law. In 2008 15th Working Conference on Reverse Engineering, pp. 101-110. IEEE, 2008.
Huangzhao Zhang, Zhuo Li, Ge Li, Lei Ma. Yang Liu, and Zhi Jin. Generating Adversarial Examples for Holding Robustness of Source Code Processing Models. Proceedings of the AAAI Conference Artificial Intelligence, 34(01):1169 1176, April 2020. ISSN 2374-3468, 2159- 5399, doi: 10.1669/aaai.v article/view/S5 v34i01.544 URL /ttp://aaaaaa.. index. ptppa////A.pppp/AAAA//
A ADVERSARIAL TRAINING
We perform adversarial training in order to make CODE2SEQ most robust to adversarial attacks. To test the robustness of adversarially-traine COdE2sEQ model, we perform the following experiment:
1. We train CODF2SFQ-SM model.
2. We generate STRATA perturbations of the java-small training dataset using vocabulary of the top-1000 subtokens by L2 norm of the embedding. For each training example, we generate five adversarial perturbations for each concatenation strategy, single, 5-diff. and 5-same.
3. We fine-tune the CODE2SEQ-SM model with the adversarial examples for a final epoch.
Baseline
single
5-diff
5-same
Clean-Tr Tr Ady-Tr
.369
.362
.263
.214
.371
.382
.370
.366
Table 6: Results of adversarial training of CODE2SEQ-SM on 1.2-based adversarial examples.
Our adversarial training results are preliminary. Further work is needed to evaluate the robustness of the model after adversarial training. Nonetheless, our results strongly suggest that adversarial training with STRATA adversarial examples highly effective at defending against STRATA attacks (Table Unexpectedly, training with adversarial examples slightly improves the ability of the model to classify clean data (improvement from Fl of 0.369 to 0.371), suggesting that the added robustness forces the model to learn better representation of the source code. This adversarial fine-tuning was not more computationally intensive than standard training and represents a simple way to make this model of code more robust.
Our method differs from standard adversarial training method proposed in Madry et al. (2018) in which adversarial examples regenerated every epoch. Since our adversarial examples only rely
11

---

Under review as a conference paper at ICLR 2021
on knowing the top-n subtokens by embedding L2 norm which does not change drastically over time, we do not have to recompute new adversarial examples after each epoch and instead can compute large batches of adversarial examples prior training. Since STRATA is gradient-free and does not require re-computation every step in the adversarial training process, our method is extremely inexpensive, whereas traditional gradient-based methods for developing adversarial examples are highly expensive, and are less effective (Table Therefore, our method can represent an extremely casy hardening technique for all types of models operating in the discrete domain with large vocabulary space. Further, we have shown that traditional gradient- based adversarial training is largely ineffective at defending against our attack (Table 4 thus our method vital for a comprehensive defense against adversarial attacks.
B FULL COMPARISON WITH OTHER CODE2SEQ ATTACKS
We have shown that STRATA works well to attack the CODE2SEQ model and can outperform the attacks by [Ramakrishnan al. 2020). Here, we present a comparison by the transformations proposed Rabin & Alipour (220)) In order to be able to compare the same metrics, we calculate the percent prediction change, which the percent of the time that an adversarial change resulted in change in prediction. A higher percent prediction change indicates a better attack.
In Table 7 we compare the performance of our attack to the performance of the transformations generated by Rabin & Alipour (2020) We find that the transformations performed by Rabii & Alipour (2020) result in fewer prediction changes than STRATA. As above, the most effective strategy is our 5-same attack.
Adversarial attack
% prediction change
Variable renaming (Rabin et al.)
47.04
Boolean Exchange (Rabin et al.)
51.43
Loop Exchange (Rabin et al.)
42.51
Permute Statement (Rabin et al.)
43.53
All (STRATA, 5-same)
77.3
Top 6k by L.2 (STRATA, 5-same) Top 10k by Frequency (STRATA, 5-same)
86.3
84.5
Table 7: Comparison of the percent of the time the prediction is changed by STRATA and by an attack from Rabin et al. Both our attack and the attack from Rabin et al. target CODE2SEQ-LG, Note that our model is evaluated on the java-small testing dataset.
C CROSS-DATASET ATTACKS
We present two fully black-box attacks that do not require any information about the targeted CODE2SEQ model or dataset:
As a surrogate model, we train CODE2SEQ model on any available dataset for the targeted pro- gramming language. To obtain adversarial examples from the surrogate, we identify optimal L2 and frequency cutoffs for this model. Using these cutoff', we construct vocabulary of the optimal top-n by frequency or by L2 norm. We show that these adversarial examples can be transferred to other models.
We present the results of the cross-dataset transfer attack proposed in Section 3.3 In particular, we generate both frequency and 1.2 STRATA adversarial examples. We use the T.2 norm of the embeddings of CODE2SEQ-SM, CODE2SEQ-MD, and CODE2SEQ-LG, and the subtoken frequencies of java-small, java-medium, and jave-large to construct six different collections adversarial examples, of which each collection is perturbation of the java-small test set. We test each dataset on each model. Tables g and show the results of the experiments, revealing that while the white-box and known-dataset attacks (the diagonals of the tables) outperform the cross-dataset attacks, the cross-datase attacks are nonetheless effective. Furthermore, we note that based cross-da dataset attacks are more effective than Trequency-based cross-dataset attacks, confirming that 1.2 norms can
12

---

Under review as a conference paper at ICLR 2021
effectively identify subtokens that are high impact in other models. We conclude that STRATA be performed in a true black-bon box setting with no information about the model parameters nor the training dataset. The cross-dataset attack is likely effective due to similar distributions of the Java datasets. Similar to word frequencies in natural language corpora, we expect that most Java datasets should have similar subtoken distributions, and thus STRATA should transfer across models trained on different datasets.
java-small (Ik, (.2)
java-medium (3k, L.2)
java-large (6k, L.2)
CODE2SEQ-SM
.214
.253
.290
CODE2SEQ-MD
.359
.349
.433
CODE2SEQ-LG
.373
.361
.358
Table 8: Fl scores on adversarial data generated by a cross-dataset attack where vocabularies are constructed by using the L.2 norm of the embeddings of the CODE2SEQ model trained on the particular dataset. The first column corresponds with the model that is being attacked, and the other columns correspond with the dataset from which the attack is constructed. Lower score means a better attack. All adversarial subtokens are concatenated with 5-sume. The boxed scores correspond with the baseline same-dataset attacks.
java-small (1.8k, freq)
java-medium (3k, freq)
java-large (10k, freq)
CODE2SEQ-SM .235
.248
.279
CODE2SEQ-MD .381
.377
.417
CODE2SEQ-LG
.397
.392
.387
Table 9: Fl scores on adversarial data generated by a cross-dataset attack where vocabularies are constructed by using the frequency of subtokens in the associated training dataset. The first column corresponds with the model that is being attacked, and the other columns correspond with the dataset from which the attack is constructed. Lower score means a better attack. All adversarial subtokens are concatenated with 5-same. The boxed scores correspond with the bascline same-dataset attacks.
D EXAMPLES OF TARGETED ATTACKS
public static int f(ListeInteger> input) int total 0; 1-0; Lcinnut.lengthh. +i)
public static int f(List<IIntaee>> input) int answer 0; for (int i<input. ++1) answer =- input[i];
total input[i];
return total;
return onswer;
(a) Predicted sum
(b) Predicted as getInt
Figure Changing local variable identifiers affects prediction. Left: Original Java method sum computes the sum of list of integers. CODE2SEQ, trained on java-large, predicts that the method name sum. Right: Perturbed Java method with the same behavior, but the local variable total has been replaced with answer, causing CODE2SEQ to misclassify this method getInt.
To illustrate the effectiveness of STRATA targeted attacks more concretely, we target particular arbitrarily-picked subtokens and measure the success rate over the entire testing set (Table T0p and find that though effectiveness can vary across different targets, the average effectiveness quite high.
D.1 COMPUTATIONALLY INEXPENSIVE
Current alternative methods for attacking models source code with comparable results involve either extensive search for optimal transformations gradient-based optimization for token
13

---

Under review as a conference paper at ICLR 2021
Target
Target
Target
Target
tournament
86.3
ftp
53.4
concat
41.5
combat
26.6
redaction
70.5
podam
45.3
eternal
37.3
thursday
23.9
outliers
62.4
wind
44.7
orderby
32.8
girth
21.5
mission
56.8
weld
42.4
reentry
29.8
16.4
Table 10: Effectiveness of targeted attacks on CODE2SEQ-LO using uncurated and arbitrarily picked subtokens high embedding vector, sorted by % success.
replacement, combination of the two (Rabin & Alipour 2020 Ramakrishnan et al. 2020 Yefet
et al. 2122). Extensive searches are inherently computationally expensive and, in the case gradient- based optimization, oftentimes require GPU for efficient implementation. STRATA, however, implemented to run quickly on even CPU-only machines. After an initial pre-processing step to mark local variable names for easy replacement which took less than five minutes on 24-core CPU-only machine, we were able to construct adversarial examples using STRATA on dataset of 20,000 within seconds. The analogous gradient-basee method proposed by Ramakrishnan al. (2020) took multiple hours on the same machine. The combined speed and effectiveness of STRATA will allow researchers quickly harden their models against adversarial attacks with efficient large-scale adversarial training.
be
14


Reviews:
[{"review_id": "fCuTWq6RM7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes STRATA, a simple adversarial attack against the code2seq model. The key idea is to replace local variable names in the input code with other randomly chosen sub-tokens with embedding vectors of relatively high L2 norms. Meanwhile, they observe that such tokens often appear frequently in the training set, thus alternatively they can simply use frequently appeared tokens as the target to perform the attacks. In this way, they can attack the model in the black-box scenario, without the knowledge of the model parameters and the training data, as long as they can roughly approximate the frequency distribution of different code sub-tokens in the training set. They evaluate their approach on code2seq models trained for the Java code, and compare with existing attacks against code2seq models. They first show that the 5-same attack, i.e., repeating a sub-token 5 times and concatenating them as the new local variable name, is the most effective attack. This attack decreases the F1 scores more compared to the baseline attack from prior work. In addition, they show that by adding STRATA adversarial examples for adversarial training, the new model becomes more robust to their proposed attacks.\n\nRobustness of models of code is an interesting topic, and the authors show that a simple and computationally inexpensive attack could degrade the performance of the code2seq model. However, this paper suffers from both the approach design and the evaluation, and I discuss the details below.\n\n1. A very straightforward defense is to anonymize the variable names. In particular, for 5-same attacks, although each sub-token itself is common, the concatenation of them doesn't seem natural, and thus a simple input preprocessing could already make the attack ineffective.\n\n2. The authors mention that they select ~40,000 test samples from the original test set to perform the attacks. In this case, when computing the F1 score, is it computed among the 40,000 test samples, or on the entire test set? For results of attacks proposed in prior work, have you evaluated them yourself, or did you quote the numbers from previous papers? I see that the baseline results are the same as shown in Ramakrishnan et al., but I don't think they limit their attacks to code with local variables, thus I would like to double-check.\n\n3. When evaluating the adversarial training and the transferability of the previous defense, they show that the previous defense does not show better robustness against STRATA. On the other hand, training with STRATA may also not improve the robustness against previous attacks. Have the authors evaluated the transferability of the model trained with STRATA adversarial examples? Meanwhile, did you train the same model to conduct attacks using STRATA and the baseline? I noticed that the F1 scores of models without perturbations are different for STRATA and the baseline attack.\n\n4. The scope of experiments is pretty limited. The authors only focus on a single model (code2seq) for one task (Java). The authors should extend the experiments to more models and/or more code-related tasks, or at least different programming languages, such as those evaluated in the code2seq paper or previous papers on attacks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aRPNYd7A8Ys", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an Adversarial attack strategy for the code2seq model.\nThe authors observed that L2 distance between pre- and post-training embedding of a token varies significantly based on the token’s frequency. Based on that they devised strategies (black and white box) to perturb the sub-token. Here, they change the local sub-token names using three heuristics. \n\nI like the simple approach to launch the adversarial attack, which shows a promising result. However, I have the following concern:\n\n-\tThe paper lacks motivation. Are there any security implications of such attacks (e.g., malware classification, etc.)? \n-\tOnly for one model (Code2Seq) the authors have tested their scheme.\n-\tDid not compare with other attacks that work on discrete domains (especially in NLP there are many attacks)\n-\tThe three strategies proposed in the paper for perturbation. However, these strategies are quite ad-hoc (e.g., why suddenly choose 5 sub-tokens).   \n- The paper said a black-box attack (even in the title), but they reply on white box information showing that L2 distance of high-frequent tokens are more.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BUl926O9E0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a gradient-free method to craft adversarial examples for Code2Seq model. Code2Seq model generates textual summary of code snippets.\n\nOverall I think authors proposed an interesting idea of an attack, however evaluation should be improved. Thus I recommend to reject paper at this point, but encourage authors to improve the paper and resubmit.\n\nStrong points:\n* Proposed novel interesting idea of gradient-free attack on source code\n* Paper is well written and easy to understand\n\nWeak points:\n* My main concern is evaluation. While authors claim that they compared to Ramakrishnan et al, in practice it looks like this comparison was performed using different baselines. Which makes results questionable.\n* The whole paper is about attacking one very specific model (Code2Seq) and it’s not clear what is the motivation of attacking this specific model (instead of some other). It’s also not clear how easy proposed method could be adapted to other tasks (which use code as an input). \n* Adversarial training is done by generating adversarial examples only one time (before the last epoch). This may lead for the model to overfit to specific adversarial examples.\n* No comparison of adversarial training with any cheaper mechanism. For example, adding random noise to embeddings during training.\n\nRecommendations on how to improve the paper:\n* Improve evaluation, add fair comparison (i.e. on the same baseline model) with other methods.\n* Show how this method could be adapted to different tasks which use code as an input and add evaluation on these tasks.\n* Update adversarial training to be consistent with what is typically done (i.e. adversarial examples generated on each training step for current state of the model) or provide motivation for the current way adversarial training is implemented.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ad8avZV9O0a", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes STRATA, a novel adversarial attack against source code models, more precisely against code2seq. The attack strategy can be applied under black- or white-box threat models, targeted or untargeted. Adversarial training is based on STRATA adversarial examples is proposed to render the models robust. Experiments are performed on Java code datasets of variable sizes.\n\nStrong points:\n- STRATA is based on the interesting observation that token frequencies and their $L_2$ norms are strongly correlated. This information provides an elegant and effective strategy for choosing attack tokens.\n- The proposed attack seems computationally inexpensive.\n- The paper covers what one would expect when proposing a new attack, including performance evaluation, study of transferability and adversarial training results.\n- The paper is clear and well-written.\n\nConcerns:\n- STRATA seems limited in some respects: it requires access to a similar code base, can only be applied for functions with local variables, only replaces variable names.\n- The proposed strategies for generating replacement tokens are trivial and easily detectable (e.g., concatenating the same token five times). The more subtle strategies are significantly less effective.\n- The proposed adversarial training strategy does not seem enough for producing a robust model: only one epoch of fine-tuning on adversarial samples is performed at the end of natural training. This is coupled with unconvincing experimental evaluations of adversarial training. As such, the paper does not seem to prove that a robust model can or has been trained using STRATA samples.\n- The notion of adversarial examples against code in the sense used in the paper seems vague. Appendix Fig. 3 shows a qualitative example of attack, where \"product\" is replaced with \"identity\" in all variable names of a function. The model (incorrectly) predicts the name of the function being related to \"identity\", not \"product\". However, that prediction seems reasonable; a human would most likely predict the same, considering that nothing in the code of the function is related to \"product\". Can this indeed be considered an adversarial example?\n- It is unclear why the comparison to previous attacks is done under different metrics depending on the attack. Moreover, why is STRATA trained on the large Java dataset, but evaluated on the small one (Appendix, Tab. 7)?\n\nReasons for score:\n\nOverall, I lean towards rejection. I like the idea behind choosing attack tokens, but the experimental results and derivation of the adversarial training strategy do not seem thorough enough. Moreover, an attack designed specifically for models against code, tested on one architecture and one programming language only seems relevant to a small part of the community.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper gives a gradient-free method for generating adversarial examples for the code2seq model of source code.

While the reviewers found the high-level objectives interesting, the experimental evaluation leaves quite a bit to be desired. (Please see the reviews for more details.) As a result, the paper cannot be accepted in the current form. We urge the authors to improve the paper along the lines that the reviews suggest and resubmit to a different venue.
Decision: Reject