Generating Adversarial Computer Programs using Optimized Obfuscations
Authors: Shashank Srikant
Sijia Liu
Tamara Mitrovska
Shiyu Chang
Quanfu Fan
Gaoyuan Zhang
Una-May O'Reilly
Venue: ICLR 2021 Poster (2021)
Keywords: Machine Learning (ML) for Programming Languages (PL)/Software Engineering (SE)
Adversarial computer programs
Program obfuscation
Combinatorial optimization
Differentiable program generator
Models for code

Abstract: Machine learning (ML) models that learn and predict properties of computer programs are increasingly being adopted and deployed. 
These models have demonstrated success in applications such as auto-completing code, summarizing large programs, and detecting bugs and malware in programs. 
In this work, we investigate principled ways to adversarially perturb a computer program to fool such learned models, and thus determine their adversarial robustness. We use program obfuscations, which have conventionally been used to avoid attempts at reverse engineering programs, as adversarial perturbations. These perturbations modify programs in ways that do not alter their functionality but can be crafted to deceive an ML model when making a decision. We provide a general formulation for an adversarial program that allows applying multiple obfuscation transformations to a program in any language. We develop first-order optimization algorithms to  efficiently determine two key aspects -- which parts of the program to transform, and what transformations to use. We show that it is important to optimize both these aspects to generate the best adversarially perturbed program. Due to the discrete nature of this problem, we also propose using randomized smoothing to improve the attack loss landscape to ease optimization. 
We evaluate our work on Python and Java programs on the problem of program summarization. 
We show that our best attack proposal achieves a $52\%$ improvement over a state-of-the-art attack generation approach for programs trained on a \textsc{seq2seq} model.
We further show that our formulation is better at training models that are robust to adversarial attacks.

Published as a conference paper at ICLR 2021
GENERATING ADVERSARIAL COMPUTER PROGRAMS USING OPTIMIZED OBFUSCATIONS
Shashank Srikant"
Sijia Liu Gaoyuan Zhang?
Tamara Mitrovska Una-May O'Reilly'
Shiyu Chang?
Quanfu Fan?
CSAILL, MIT MIT-IBB Watson Al Lab shashêmit edu, liusi jirijii5maauu edu, mnamay@csail mit. edu
3 Michigan State University
ABSTRACT
Machine learning (ML) models that learn and predict properties of computer pro- grams are increasingly being adopted and deployed. In this work, we investi- gate principled ways to adversarially perturb a computer program to fool such learned models, and thus determine their adversarial robustness. We use program obfuscations, which have conventionally been used to avoid attempts reverse engineering programs, as adversarial perturbations. These perturbations modify programs in ways that do not alter their functionality but can be crafted to deceive an ML model when making a decision. We provide a general formulation for an adversarial program that allows applying multiple obfuscation transformations to a program in any language. We develop first-order optimization algorithms to ef- ficiently determine two key aspects which parts of the program to transform, and what transformations to use. We show that it is important to optimize both these aspects generate the best adversarially perturbed program. Due the discrete nature of this problem, we also propose using randomized smoothing to improve the attack loss landscape to ease optimization. We evaluate our work on Python and Java programs on the problem of program summarization [ We show that our best attack proposal achieves a 52% improvement over a state-of-the- attack generation approach for programs trained on SEQ2SEQ model. We further show that our formulation is better at training models that are robust to adversarial attacks.
INTRODUCTION
Machine learning (ML) models are increasingly being used for software engincering tasks. Appli- cations such as refactoring programs, auto-completing them in editors, and synthesizing GUI code have benefited from ML models trained on large repositories of programs, sourced from popular websites like GitHub (Allamanis et al. 2018). They have also been adopted to reason about and assess programs (Srikant& & Aggarwal 2014 Si et al. 2018). find and fix bugs (Gupta et al. 2017 Pradel & Sen 2018), detect malware and vulnerabilities in them (Li et al. L
Zhou et al. 2019
etc. thus complementing traditional program analysis tools. As these models continue be adopted for such applications, it important to understand how robust they are adversarial attacks. Such attacks can have adverse consequences, particularly in settings such as security (Zhou et al. 2019) and compliance automation (Pedersen 2010). For example, an attacker could craft changes in ma- licious programs in way which torces model to incorrectly classify them being benign, or make changes to pass off code which licensed as open-source in an organization's 's proprietary code-base base.
Adversarially perturbing a program should achieve two goals trained model should flip its deci-
sion when provided with the perturbed version of the program, and second, the perturbation should be imperceivable. Adversarial attacks have mainly been considered in image classification (Good-
fellow et al., 2014; [Carlini & Wagner 2017; Madry et al. 2018), where calculated minor changes made to pixels of an image are enough to satisty the imperceptiiliity requirement. Such changes escape human's 's attention by making the image look the same as before perturbing it, while modify- ing the underlying representation enough flip classifier's ss decision. However, programs demand a stricter imperceptibility requirement not only should the changes avoid human attention, but the changed program should also importantly functionally behave the same as the unperturbed program.
Source code: http:////tttabboooo//////gggrpes a code wwwe-ooodeeeeeeeeeee generation

---

Published as a conference paper at ICLR 2021
Program obfuscations provide the agency to implement programs. Obfuscating computer programs have long been used as a way to avoid attempts at eweers-engiineering them. They transform a program in a way that only hampers humans' com- prehension of parts of the program, while retaining its original semantics and functionality. For example, one common obfuscation operation is rename variables in an attempt to hide the pro- gram's intent from a reader. Renaming a variable sum in the program statement int sum 0 to int xy2 = O neither alters how a compiler analyzes this variable nor changes any computations or states in the program; it only hampers our understanding of this variable's role in the program. Modifying a very small number of such aspects of program marginally affects how we compre- hend it, thus providing way to produce changes imperceivable to both humans and a compiler. In this work, we view adversarial perturbations to programs as a special case of applying obfuscation transformations to them.
such set of imperceivable changes in
Having identified set of candidate transformations which produce imperceivable changes. specific subset needs to be chosen in a way which would make the transformed program ad adversarial. Recent attempts (Yefet et al. 2019 Ramakrishnan et al. 2020 Bielk & Vechev 2020) which came closest to addressing this problem did not offer any rigor- ous formulation. They recommended using a vari- ety of transformations without presenting any prin- cipled approach to selecting an optimal subset of transformations. We present a formulation which when solved provides the exact location to trans- form as well transformation apply at the loca- tion. Figure [ iillustrates this. A randomly selected local-variable (name) when replaced by the name which is generated by the state- of-the-art attack generation algorithm for programs (Ramakrishnan et al., 2020). is unable to fool a program summarizer (which predicts set item) unless our proposed site optimization is applied. We provide a detailed comparison in Section 2 In our work, we make the following key contributions
Unperturbed
def _.w.kemmm_...e rame, vel:
name, vall forbid multi line headers/ommee eome, vil selfencociina MIME hed aas swittem buil rentr, well
Prediction:
Random site-selection; Optimal site pett--errubbbinnnoo mursinstannt 2009
def sew.teemmm..ee...ee Bel, name, vol:
virtualname, val bid must liw haaders/aume val, www.uanccddnngg
WIME sariem Joell virtuainame, vii) Prediction: set item
Optimal site-selaction; Optimal site-perturbatton mis
def soteeteemmmmee giere voll:
name, val fortid mutti wee..ineeeeeea... www.henddrssggsee..oooo vo. selfencoddin)) MIME MM.. sutitam bull name, vall
Prediction: watte
The advantage of our formulation when compared to the statc-of-the-art.
Figure 1:
vi sirtualnne
We identify two problems central defining an adversarial program identifying the sites in a program to apply perturbations on, and the specific perturbations to apply on the selected sites. These perturbations are involve replacing existing tokens or inserting new ones.
We provide general mathematical formulation of perturbed program that models site loca- tions and the perturbation choice for each location. It independent of programming languages and the task on which a model is trained, while seamlessly modeling the application of multiple transformations to the program.
We propose a set of first-order optimization algorithms solve our proposed formulation effi- ciently, resulting in a differentiable generator for adversarial programs. We further propose a randomized smoothing algorithm to achieve improved optimization performance.
Our approach demonstrates a 1.5x increase in the attack success rate over the state-of the-rrr attack generation algorithm (Ramakrishnan et al. 2020) on large datasets of Python and Java programs.
We further show that our formulation provides better robustness against adversarial attacks com- pared to the ofate-of-the-ar when used in training an ML model.
2 RELATED WORK
Due to large body of literature on adversarial attacks in general, we focus on related works in the domain of computer programs. Wang & Christodorescu (2019), Quiring et al.) (2019). Rabin et al. (2020), and Pierazzi et al. (2020) identity obfuscation transtormations as potential adversarial examples. They do not, however, find an optimal set of transformations to deceive a downstream model. [Lu et al. (2017) provide a stochastic optimization formulation to obfuscate programs op- timally by maximizing its impact on an obscurity language model (OLM). However, they do not address the problem of adversarial robustness of MI. models of programs, and their formulation is only to find the right sequence of transformations which increases their OLM's perplexity. They use an MCMC-hased search to find the best sequence.
Yefet et al. (2019) propose perturbing programs by replacing local variables, and inserting print statements with replaceable string arguments. They find optimal replacements using first-order optimization method, similar to Balog et al. (2016) and HotFlip (Ebrahimi et al. 2017). This is
2

---

Published as a conference paper at ICLR 2021
def fool }: =
def foo( ):
def foo )
def
I1 print(+)
world
r
b = O I2 print(+)
world
a 9. get, set, hello, world
print("set")
= b
def fool je bF
fool print(
Prediction: foo
I3 print(*)
Prediction: delete
b 5
print )
r= b+5
[a]
(b)
(c)
Figure 2: (a) A sample program P containing a function foo (b) P contains five sites which can be trans-
formed two replace sites corresponding to local variables b and and threc insert sites at locations 1 11, 12, 13. Q is vocabulary of tokens which can be used for the transformations. (c) This is a perturbed program with the tokens world and set from f used to replace tokens b and at location I3. These transformations do not change the original functionality of P. but cause an incorrect prediction delete (d) Examples of two site selection vectors z', x² selecting different components. z; 1 for location signifies that the ith token in P is selected to be optimally transformed. z corresponds to the perturbed program in (c).
an improvement over Zhang et al. (2020), who use the Metsopolis-Hastings algorithm to find an optimal replacement for variable names, Bielik & Vechev] (2020) propose a robust training strategy which trains a model to abstain from deciding when uncertain if an input program is adversarially perturbed. The transformation space they consider is small, which they search through greedily. Moreover, their solution is designed to reason over a limited context of the program (predicting variable types), and is non-trivial to extend to applications such program summarization (explored in this work) which requires reasoning over an entire program.
Ramakkiihhan et (2020) extend the work by Yefet al. (2019) and is most relevant to what we propose in this work. They experiment with a larger set of transformations and propose a standard min-max formulation to adversarially train robust models. Their mner-maximizer which generates adversarial programs, models multiple transformations applied to a program in contrast to Yefet et al. (2019). However, they do not propose any principled way to solve the problem of choosing between multiple program transformations. They randomly select transformation operations to apply, and then randomly select locations in the program to apply those transformations on.
We instead show that optimizing for locations alone improves the attack performance. Further, propose joint optimization problem of finding the optimal location and optimal transformation, only the latter which RRaaakiihhann et al. (2020) (and Yefet et al. (2019) address in a princi- pled manner. Although formally unpublished at the time of preparing this work, we compare our experiments to |Ramakrishnan et al. 2020), the state-of-the-art in evaluating and defending against adversarial attacks on models for programs, and contrast the advantages of our formulation.
3 PROGRAM OBFUSCATIONS AS ADVERSARIAL PERTURBATIONS
In this section, we formalize program obfuscation operations, and show how generating adversarial programs can be cast as constrained combinatorial optimization problem.
Program obfuscations. We view obfuscation transformations made to programs as adversarial per- turbations which can affect downstream ML/DL model like malware classifier
program summarizer. While a variety of such obfuscation transformations exist for programs in general (see section 2A, Liu et al. (2017)). we consider two broad classes replace and insert transformations. In replace transtormations, existing program constructs are replaced with variants which decrease readability. For example, replacing variable's name, function parameter's 's name, or object field's name does not affect the semantics the program any way. These names in any program exclusively aid human comprehension, and thus serve as three replace transformations. In insert transformations, we insert new statements to the program which are unrelated to the code it is in- serted around, thereby obfuscating its original intent. For example, including print statement with an arbitrary string argument does not change the semantics of the program in any way.
Our goal hence is to introduce a systematic way to transform a program with insert or replace transformations such that a trained model misclassifies program P that it originally classified correctly.
3

---

Published as a conference paper at ICLR 2021
Site-selection and Site-perturbation Towards defining adversarial programs. Before we for- mally define an adversarial program, we highlight the key factors which need to be considered in our formulation through the example program introduced in Figure
Consider applying the following two obfuscation transformations on the example program P in Figure -a - reppaainn local variable names (a replace transform), and inserting print statements (an insert transform). The two local variables and in are potential candidates where the replace transform can be applied, while a print statement can potentially be inserted at the three locations I1, I2, 33 (highlighted in Figure bb.. We notate these choices in a program as sites- locations in a program where unique transformation can be applied.
Thus, in order to adversarially perturb P. we identify two important questions that need to be ad- dressed. First, which sites in a program should be transformed? Of the n sites in program, if we are allowed to choose at most k sites, which set of S k sites would have the highest impact on the downstream model's performance? We identify this as the site-selection problem, where the con- straint & the perturbation strength of an attacker. Second, what tokens should be inserted/replaced at the k selected sites? Once we pick k sites, we still have to determine the best choice of tokens to replace/insert at those sites which would have the highest impact on the downstream model. We refer to this as the sile-perturbation problem.
Mathematical formulation. In what follows, propose general and rigorous formulation of adversarial programs. Let P denote a benign program which consists of a series of n tokens {P;) n in the source code domain. For example, the program in Figure Da a. when read from top to bottom and left to right, forms a series of n 12 tokens {def, b. I. 5}. We ignore white spaces and other delimiters when tokenizing. Each P: {0, 1,101 here is considered one-hot vector of length |92], where 92 is a vocabulary of tokens. Let P' define a perturbed program (with respect to P) created by solving the site-selection and site-perturbation problems, which use the vocabulary Ω to find an optimal replacement. Since our formulation is agnostic to the type of transformation, perturbation in the remainder of this section refers to both replace and insert transforms. In our work, we use a shared vocabulary Q select transforms from both these classes. In practice, we can also assign a unique vocabulary to each transformation we define.
To formalize the site-selection problem, we introduce a vector of boolean variables z E {0, 11" to indicate whether or not a site is selected for perturbation. If 2, = 1 then the ith site (namely, P.) is perturbed. If there exist multiple occurrences of token in the program, then all such sites are marked praaraaiaaa
1. For example, in Figure d if the site corresponding to local variable indices of its occurrences, 23, 2n marked as 1 as shown in 2!. Moreover, the number of perturbed sites, namely, 1 < k provides means of measuring perturbation strength. For example, k is the minimum perturbation possible, where only one site is allowed to be perturbed. To define settrpeettrbation, we introduce one-hot vector u, {0, from Q which would serve as the insert/replace token for chosen transformation at a chosen site. If the jth entry [ul] 1 and 2; 1, then the jth token in 52 is used as the obfuscation transformation applied at the site i (namely, to perturb P.). We also have the constraint 1 u, = 1, implying that only one perturbation is performed at P.. Let vector E {0, 1) 1|nx|(() denote n different u, vectors, one for each token in P.
selected, then both
encode the selection of token
Using the above formulations for site-selection, site-perturbation and perturbation strength, the perturbed progrum P' can then be defined as
p' = (1 P +2 u, where 1 2 S k. z E (0, 1'u, = 1, u E 10, R Vi, where denotes the element-column wise product.
(1)
The adversarial effect of P' is then measured by passing it as input to a downstream ML/DL model 0 and seeing if it successfully manages to fool it.
Generating a successful adversarial program is then formulated as the optimization problem,
minimize Eastact ((1 2) P +2 12p.2uuuPP.. P. s.u
subject constraints in |.
where lattack denotes an attack loss. In this work, we specify Lattack as the cross-entropy loss on the predicted output evaluated at P' in an untargeted setting (namely, without specifying the prediction label targeted by an adversary) (Ramakrishnan et al.) (202)). One can also consider other specifications of fataakk e.g., C&W untargeted and targeted attack losses (Carlini & Wagner 2017).
4

---

We use the randomized Published as a conference paper at ICLR 2021
4 ADVERSARIAL PROGRAM GENERATION VIA FIRST-ORDER OPTIMIZATION
Solving problem is not trivial because of its combinatorial nature (namely, the presence of boolean variables), the presence of bi-linear objective term (namely, 2 u), as well as the presence of multiple constraints. To address this, we present a projected gradient descent (PGD) based joint optimization solver (JO) and propose alternates which promise better empirical performance.
PGD as a joint optimization (JO) solver. PGD has been shown be one of the most effective attack generation methods to fool image classification models (Madry et et. 2018). Prior to applying PGD, instantiate 2 into feasible version by relaxing boolean constraints to their convex hulls,
minimize 2,0
Pattack(z, u)
subject to 5 E [0,
= 1'
1'u,
E [0.1] Vi,
where for ease of notation, the attack loss in (2) is denoted by fatrack (z, u). The continuous relax- ation of binary variables in (3) is commonly used trick combinatorial optimization boost the stability of learning procedures in practice (Boyd et al.] (004). Once the continuous optimization problem (3) is solved, a hard thresholding operation or randomized sampling method (which re- gards z and as probability vectors with elements drawn from Bernoulli distribution) can be called to map a continuous solution its discrete domain (Blum& & Roli (003). sampling method in our experiments.
The PGD algorithm is then given by
(z
(t-1) u (t 1) af V (2 " (11)),,, Vulautac(2 "
(4)
(z } (roj((z()) u (
(5)
where denotes PGD iterations, (0)) and (o) are given initial points, a > 0s a learning rate, V, denotes the first-order derivative operation w.r.t. the variable z, and Proj represents the projection operation w.r.t. the constraints of (3.
The projection step involves solving for z and u, simultaneously in complex convex problem. See Equation 9 in Appendix for details.
key insight that the complex projection problem ( can equivalently be decomposed into sequence of sub-problems owing to the separability of the constraints w.r.t. z and {u:}. The two sub-problems are
minimize a subject 1" < k, E [0,
||z - 1
minimize Ty;
||u. -lu- u 1
and
Vi.
(6)
subject to 1Tu = U 10
The above subproblems w.r.t. z and u; can optimally be solved by using a bisection method that finds the root of a scalar equation. We provide details of closed-form solution and its corresponding proof in Appendix A We use this decomposition and solutions to design alternating optimizer, which we discuss next.
Alternating optimization (AO) for fast attack generation. While JO provides an approach to solve the unified formulation in (2). it suffers from the problem of getting trapped poor local optima despite attaining stationarity (Ghadimi al. (0066). We propose using AO (Bezdek & Hath- away 2003) which allows the loss landscape to be explored more aggressively, thus leading to better empirical convergence and optimality (see Figure|4a).
AO solves problem (2) one variable time first, by optimizing the site selection variable keeping the site perturbation variable u fixed, and then optimizing keeping z fixed. That is,
arg min
lactace (z, (1)) and u
arg min lattack (2 u) Vi.
(7)
Tuugl 1]([ We can use PGD, as described in (6). to similarly solve each of and separately in the two alternating steps. Computationally. AO expensive than JO by a factor of 2, since we need two iter- ations to cover all the variables which JO covers in single iteration. However, in our experiments, find AO converge faster. The decoupling in AO also eases implementation, and provides the flexibility to set a different number of iterations for the u-step and the z-step within one iteration of AO. We also remark that the AO setup in (7) can be specified in other forms, e.g. alternating direc- tion method of multipliers (ADMM) (Boyd et al.] (01T). However, such methods use an involved alternating scheme to solve problem We deter evaluating these options to future work.
IT*kk, xE (o.
5

---

Published as a conference paper at ICLR 2021
Randomized smoothing (RS) to improve generating adversarial programs. In our experi- ments, we noticed that the loss landscape of generating adversarial program is not smooth (Figure 3). This motivated us to explore surrogate loss functions which could smoothen it out. In our work, we employ convolution-basee ficulty induced by the non non-smoothness of the attack loss loss smoothh
RS technique (Duchi et al. 2012) to circumvent the optimization dif-
attack- We eventually obtain a smoothing
temooth (z, u) E μe u (T)),
(8)
where & and T are random samples drawn from the uniform distribution within the unit Euclidean ball, and y is a small smoothing parameter (set 0.01 our experiments).
The rationale behind RS (8) is that the convolution of two Tunctions (smooth probability density function and non-smooth attack loss) is at least as smooth as the smoothest of two original functions. The advan- tage of such formulation is that it independent of the loss function, downstream model, and the optimiza- tion solver chosen for a problem. We evaluate RS on both AO and JO. In practice, we consider an empirical Monte Carlo approximation of (8) (attack
0
622
-021
-0.4
(a)
(b)
Samooth (z, u) μE,, u+ We set experiments to save on computation time. We also find that smoothing the site perturbation variable tributes the most to improving attack performance. We hence perturb only u to further save computation time.
10 in our
Figure 3: The original loss landscape for a sample program
Bap. Randomized smoothing produces flatter smoother loss landscape (bb.. plot the loss along the space determined by the vector (..sgn(V, [(x)] + 3.Rademacher(O... )) for a.
1-0.05, 0.05] (Engstrom al. 2018
5 EXPERIMENTS & RESULTS
We begin by discussing the following aspects of our experiment setup 10 the classification task, the dataset and model we evaluate and the evaluation metrics we use.
Task, Transformations, Dataset. We evaluate formulation of generating optimal adversarial programs on the problem of program summarization, first introduced by Allamanis al. (2016). Summarizing a function in a program involves predicting its name, which is usually indicative of
intent. We use this benchmark to test whether our adversarially perturbed program, which retains the functionality of the original program, can force a trained summarizer predict incorrect function name. We evaluate this on well maintained dataset of roughly 150K Python eet al. 2016) and 700K Java programs (Alon et al. 2018). They are pre-processed into functions. and each function is provided as input an ML model. The name of the function is omitted from the input. The ML model predicts sequence of tokens the function name. We evaluate our work on six transformations (4 replace and insert transformations): see Appendix [or details on these transformations. The results and analysis that follow pertains to the case when any of these six transformations can be used as a valid perturbation, and the optimization selects which to pick and apply based on the perturbation strength k. This is the same setting employed in the baseline Ramakkishhan et al. 2020).
Model. We evaluate a trained SEQ2SEQ model. It takes program tokens as input, and generates sequence of tokens representing its function name. We note that our formulation is independent of the learning model, and can be evaluated any model for any task. The SEQ2SEQ model is trained and validated on 90% of the data while tested on the remaining 10%. It optimized using the cross-entropy loss function.
CODE2SEO (Alon et al. 2018) is another model which has been evaluated on the task of program summarization. Its architecture is similar that of SEQ2SEQ and contains two encoders one which encodes tokens, while another which encodes AST paths. The model when trained only on tokens performs similar to model trained on both tokens and paths (Table 3, Alon et al. (018)). Thus adversarial changes made to tokens, as accommodated by our formulation, should have high impact the model's output. Owing to the similarity in these architectures, and since

---

Published as a conference paper at ICLR 2021
h:
1 site
k 5 sites
Method
ASR
F1
ASR
F1
No attack
0.00
100.00
0.00
100.00
Random replace
0.00
100.00
0.00
100,00
BASELINE
19.87
78.18
37.50
59.54
AO JO
23.16
+3.29
74.78
-3.40
43.53
+6.03 A
53.75
-5.79 A
23.32
+3,45
74.56
-3.62
41.95
+4,45 A
56.06
-3,48 A
AD RS
30.25
+10.38
69.52
-8.66
51.68
+14.18
47.92
-11.62 A
JO + RS 23.95
+4.08
74.24
-3.94
48.70
+11.20
51.55
-7.99
Table 1: Our work solves two key problems find optimal adversarial perturbations site-selection and site-perturbatin. The BASELINE method refers to [Ramakrishnan et al.] (220)). The perturbation strength is the maximum number of sites which an attacker can perturb. Higher the the Attack Success Rate (ASR), better the attack; the converse holds for F1 score. Our formulation (Eq. [ solved using two methods alternate optimization (AO) and joint optimization (JO), along with randomized smoothing (RS), shows a consistent improvement in generating adversarial programs. Differences in ASR, marked in blue, are relative to BASELINE. The results on Java dataset are tabulated in Table Appendix.
computational bench is in Pytorch while the original CODE2SEQ implementation is in TensorFlow, we defer evaluating the performance of our formulation on CODE2SEQ to future work.
Evaluation metrics. We report two metrics Attack Success Rate (ASR) and Fl-score. ASR is defined as the percentage of output tokens misclassified by the model on the perturbed input but correctly predicted on the unperturbed input, iee. ASR = Σ., ((0()))))) for each token j in the Eis 1(0(x))=Ms))
expected output of sample i. Higher the ASR, better the attack. Unlike (Ramakrishnan et al. 2020), we evaluate our method on those samples in the test-set which were fully, correctly classined by the model. Evaluating on such fully correctly classified samples provides direct evidence of the adversarial effect of the input perturbations (also the model's adversarial robustness) by excluding test samples that have originally been misclassified even without any perturbation. We successfully replicated results from (Ramakrishnan et al. (2000) on the Fl-score metric they use, and acknowl- edge the extensive care they have taken ensure that their results are reproducible. As reported in Table 2 of (Ramakrishhan et al.] (020), a trained SEQ2SEQ model has an Fl-score of 34.3 evaluated on the entire dataset. We consider just those samples which were correctly classified. The Fi-score corresponding to 'No attack' in Table i hence 100. In all, we perturb 2800 programs in Python and 2300 programs in Java which correctly classified.
5.1 EXPERIMENTS
We evaluate our work in three ways - first, we evaluate the overall performance of the three ap- proaches we propose AO, JO. and their combination with RS, to find the best sites and pertur- bations for given program. Second, our optimizers the number of iterations they are evaluated on, and the perturbation strength (k) of an attacker. Third, we use our formulation to train an adversarially robust SEQ2SEQ model, and evaluate its performance against the attacks we propose.
evaluate the sensitivity of two parameters which control
overall results. The first row corresponds to the samples not being perturbed at all. The ASR as expected is O. The 'Random replace' in row 2 corresponds to both z and u being selected at random. This produces no change in the ASR, suggesting that while obfuscation transformations can potentially deceive ML models, any random transformation will have little effect. It is important to have some principled approach to selecting and applying these transformations.
Overall attack results. Table I summarizes
Ramakrishnan et al. 2020) (referred to as BASELINE in T evaluated their work in two settings. In the first setting, they pick site at random and optimally perturb it. They refer to this as 2 We contrast this by selecting an optimal site through our formulation. We use the same algorithm as theirs to optimally perturb the chosen site i.e. to solve for u. This allows us to ablate the effect of incorporating and solving the site-selection problem in our formulation. In the second related setting, they pick sites at random and optimally perturb them, which they refer to as Øg In our setup, O and 23: are equivalent to setting ào = and h = respectively, and picking random sites in z instead of optimal ones. We run AO for 3 iterations, and JO for 10 iterations.
We find that our formulation consistently outperforms the baseline. For h: 1, where the at- tacker can perturb at most 1 site, both AO and JO provide 3 point improvement in ASR, with
7

---

Published as a conference paper at ICLR 2021
JO marginally performing better than AO. Increasing l improves the ASR across all methods the attacker now has multiple sites to transform. For = where the attacker has at most sites to transform, we find AO to provide 6 point improvement in ASR over the baseline, outperforming JO by 1.5 points.
Smoothing the loss function has marked effect. For k 1, we find smoothing to provide 10 point increase (~ 52% improvement) in ASR over the baseline when applied to AO, while JO+RS provides a point increase. Similarly, for k = 5, we find AO+RS to provide a 14 point increase (~ 38% improvement), while JO+RS provides a 11 point increase, suggesting the utility of smoothing the landscape to aid optimization.
Overall, we find that accounting for site location in our formulation combined with having smooth loss function to optimize improves the quality of the generated attacks by nearly 1.5 times over the state-of-the-art attack generation method for programs.
Effect of solver iterations and perturbation strength k. We evaluate the attack performance (ASR) of our proposed approaches against the number of iterations at À 5 (Figure 4a). For comparison, we also present the performance of BASELINE, which not sensitive to the number of iterations (consistent with the empirical finding in (Ramakrishnan et al., (020)), implying its least optimality. Without smoothing. JO takes nearly 10 iterations to reach its local optimal value, whereas AO achieves it using only 3 iterations but with improved optimality (in terms of higher ASR than JO). This supports our hypothesis that AO allows for a much more aggressive exploration of the loss landscape, proving to empirically outperform JO. With smoothing, we find both AO+RS and JO+RS perform better than AO and JO respectively across iterations. We thus recommend using AO+RS with 1-3 iterations as an attack generator to train models that are robust such ad adversarial attacks.
In Figure 4b we plot the per-
sites
100
formance of the best perform-
arna success Fatz (oSR)
100
Jo
AD
Attack Success Rate (ASR)
Baseline AO
RS
ing methods as attacker's 's perturbation strength
vary the
80
Beseline
RS
70
RS
(k). We make two key obser- vations First, allowing few sites (< 5) to be perturbed is enough to achieve 80% of the best achievable attack rate. For example, under the AO+RS at- tack, the ASR is 50 when k = and 60 when = 20. From an attacker's perspective, this makes it convenient to effec- tively attack models of programs without being discovered. Second, we observe that the best performing methods we propose con-
50
50
30
20
10
5
10 Beeations
15
20
10
15
20
Perturbation Strength 1k)
(a)
(b)
Figure 4: ASRs of our approaches and BASELINE against the number
of optimization iterations 4a) and perturbution strength of an attacker (4b).
sistently outperform BASELINE across different k. The performance with BASELINE begins to converge only after k = 10 sites, suggesting the effectiveness of our attack.
Improved adversarial training under proposed attack. Adversarial training (AT) (Madry et al.) 2018) is min-max optiiization based training method to improve model's adversarial robust- ness. In AT, an attack generator is used as the in- ner maximization oracle to produce perturbed train- ing examples that are then used in the outer min- imization for model training. Using AT, in- vestigate if our proposed attack generation method (AO+RS) yields an improvement in adversarial ro- bustness (over BASElINE) when it is used to adver- sarially train the SEQ2SEQ model. We evaluate AT in three settings 'No AT,, corresponding to the regularly trained SEQ2SEQ model (the one used in all the experiments in Table T]. BASELINE SEQ2SEQ trained under the attack by (Ramakrishnan et al. 2020). and AO+RS SEQ2SEQ trained under our AO+RS attack. We use three attackers on these models BASELINE and two of our strongest attacks AO and AO+RS. The row corresponding to 'No AT' the same as the entries under l in Table I We find AT with BASELINE improves robustness by ~1 points under AO+RS, our strongest attack. However, training with AO+RS pro- vides an improvement of 16 points. This suggests AO+RS provides better robustness to models when used as an inner maximizer in an AT setting.
Attack (ASR)
Train
BASELINE
AO
AD+RS
AT
19.87
23.16
30.25
BASELINE
17.99
18.87
19.11
AD+RS
12.73
13.01
13.75
Table 2:
We employ an AT setup to train SEQ2SEQ with the attack formulation we propose. Lower the ASR, higher the robustness to adversar- ial attacks. Training under AO+RS attacks pro- vides best robustness results.
8

---

Published as a conference paper at ICLR 2021
CONCLUSION
In this paper, we propose general formulation which mathematically defines an adversarial attack on program source code. We model two key aspects in our formalism location of the transforma- tion, and the specific choice of transformation. We show that the best attack is generated when both these aspects optimally chosen. Importantly, set up which models these two aspects is decomposable via alternating optimization. The nature of decomposition enables us to easily and quickly generate adversarial programs. Moreover, we show that randomized smoothing strategy can further help the optimizer to find better solutions. Even- tually, we conduct extensive experiments from both attack and defense perspectives demonstrate improvement our proposal over the state-of-the-an attack generation method.
identify that the joint optimization problem
ACKNOWLEDGMENT
This work was partially funded by grant by MIT Quest for Intelligence, and the MIT-IBM AI lab. We thank David Cox for helpful discussions on this work. We also thank Christopher Laibinis for his constant support with computational resources. This work was partially done during an internship by Shashank Srikant at MIT-IBM AI lab.

---

Published as a conference paper at ICLR 2021
REFERENCES
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International conference on machine learning, pp. 2091-2100,
2016.
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 113) 2018.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from struc- tured representations of code. arXiv preprint arXiv: 01400, 2018.
Matej Balog, Alexander L Gaunt, Mare Brockschmidt. Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:/611.01990 2016.
James C Bezdek and Richard J Hathaway, Convergence of alternating optimization. Neural, Parallel & Scientific Computations, 114):355,,368 2003.
Pavol Biclik and Martin Vechev. Adversarial robustness for code. arXiv preprint 27X1::2002.04699, 2020.
Christian Blum and Andrea Roli. Metaheuristics in combinatorial optimization: Overview and conceptual comparison. ACM computing surveys (CSUR), 35(3);268-308, 2003.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge uni- versity press, 2004.
Stephen Boyd, Neal Parikh, and Eric Chu. Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc, 2011.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
John C Duchi, Peter Bartlett, and Martin Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674-701, 2012.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv: 2017.
Logan Engstrom, Andrew Ilyas, and Anish Athalye. Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint arXiv: /0272, 2018
Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation meth- ods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1-2): 267-305, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 14X2v::4/2.6577 2014.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix: guage errors by deep learning. In Proceedings of the Thirly-First AAAI Conference on Artificial Intelligence, pp. 1345-1351, 2017.
Fixing common lan-
Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.0166) 2018
Han Liu, Chengnian Sun, Zhendong Su, Yu Jiang, Ming Gu, and Jiaguang Sun. Stochastic opti- mization of program obfuscation. In 2017 IEEF/ACM 39th International Conference on Software Engineering (ICSE), pp. 221-231. IEEE, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3): 127-239, 2014.
10

---

Published as a conference paper at ICLR 2021
Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, Carlos Noguera, and Lionel Seinturier. Spoon: A library for implementing analyses and transformations of java source code. Software: Practice and Experience, 46(9):1155-11 119--11999,1,, 2016.
Palle Martin Pedersen. Methods and systems for identifying an area of interest in protectable content, September 14 2010. US Patent 7,797,245.
Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. Intriguing proper- ties of adversarial ml attacks in the problem space. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1332-1349. IEEE, 2020.
Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based bug detection. Proceedings of the ACM Programming Languages, 2(00PSLA):1 2018.
Erwin Quiring, Alwin Maier, and Konrad Rieck. Misleading authorship attribution of source code using adversarial learning. In 28th (USENIX) Security Symposium ((USENIX) Security 19), pp- 479,496.20 2019.
Md Rabin, Rafiqul Islam, and Mohammad Amin Alipour. Evaluation of generalizability of neural program analyzers under semantic- semantie-preserving transformations. arXiv preprint arXiv:2004.07313 2020.
Goutham Ramakrishnan, Jordan Henkel, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. Semantic robustness of models of source code. arXiv preprint arXiv:2002.03044, 2020.
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, 51(10):731-74, 2016.
Xujic Si, Hanjun Dai, Mukund Raghothaman, Mayur Naik, and Le Song. Learning loop invariants for program verification. In Neural Information Processing Systems, 2018.
Shashank Srikant and Varun Aggarwal. A system to grade computer programming skills using ma- chine learning. In Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 1887-1896, 2014.
Ke Wang and Mihai Christodorescu. Coset: A benchmark for evaluating neural program embed- dings. arXiv preprint arXiv: /905. 1445, 2019.
Kaidi Xu, Hongge Chen, Sijia Liu, Pin-Yu Chen, Tsui-Wei Weng, Mingyi Hong, and Xue Lin. Topology attack and defense for graph neural networks: An optimization perspective. arXiv preprint arXiv:1906.042111 2019.
Noam Yefet, Uri Alon, and Eran Yahav. Adversarial examples for models of code. arXiv preprint arXiv:1910.07517, 2019.
Huangzhao Zhang, Zhuo Li, Ge Li, Lei Ma, Yang Liu, and Zhi Jin. Generating adversarial examples for holding robustness of source code processing models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1 1169-1177 176, 2020.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnera- bility identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems, pp. 10197-10207, 2019.

---

Published as a conference paper at ICLR 2021
A SOLVING THE PAIR OF PROJECTION SUB-PROBLEMS IN EQ. 6
The projection step ( can formally be described as finding the solution of the convex problem
minimize
ZZ - 1 Σ; u, u 1
(9)
subject to k, 2 E [0, 1]", 1_ 1, u, E [0, 1] Ví.
In Equation 6 we proposed to decompose the projection problem on the combined variables z and u; into the following sub-problems
minimize
||z z 1
and minimize
|| m (1) 2 u1 = 1, u, 10,
Vi.
(10)
subject to 2 S k, 1]" subject to
The following proposition is closed-form solution which solves them.
Proposition 1 Let (((11) and {u: ( denote solutions of problems given in 6 Their expressions are given by
(1+1)_ (z)
[ and ps the root of 1 (T(z(() /al]+ 1;
(11)
(1+1) Po [u (0)
P1'PPo.11 ju; (t)
1 k.
Po.1/1 T.1 3r > S.I. P'o,,. - -;11= Vi,
(12)
where []+ = max denotes the (elementwise) non-negative operation. and (-) is the (el- ementwise) box projection operation, namely, for scalar x P() x if E [0, 1], if 0, and 1 if 1.
Proof. We first reformulate problems in (6) as problems subject to a single inequality equality
constraint. That is, min ||z ( || + I]o.1]-(2) and min1 ||u u 1 112 + (o (u,), where IP(P) denotes indicator function over the constraint set C and Ic (P) if E C and co otherwise. We then derive Karush BaarushKuuhn- Kuhn-Tucker (KKT) conditions of the above problems for their
optimal solutions; see (Parikh & Boyd 2014 Xu et al. 2019) for details.
In 11 and (12). we need call an internal solver to find the root of a scalar equation, e.g., (2 ul in T1 This can efficiently be accomplished using the bisection method in the logarithmic rate O (-logg E) for the solution of 6-error tolerance (Boyd et al. 2004). Eventually, the PGD-based
JO solver formed by (4 (5). 11 and 12
12

---

Published as a conference paper at ICLR 2021
B PROGRAM TRANSFORMATIONS
To compare our work against the state-of-the-art (Ramakrishnan al. 2020). we adopt the transfor- mations introduced in their work and apply our formulation to them. This allows to solve the same setup they try while contrasting the advantages of our formulation.
They implement the following transformations.
Renaming local variables. This is the common obfuscation operation, where local variables in scope are renamed to less meaningful names. We use this to replace with a name which has an adversarial effect on a downstream model.
Renaming function parameters. Similar renaming variable names, function parameter names are also changeable. We replace names of such parameters with
Renaming object fields. A class' referenced field is renamed in this case. These fields are referenced using self and this in Python and Java respectively. We assume that the class definitions corresponding to these objects are locally defined and can changed to reflect these new names.
Replacing boolean literals A boolean literal (True, False) occurring the program replaced with an equivalent expression containing optimizable tokens. This results statement of the form <token>> = <token> and <token> != <token> for True and False respectively.
Inserting print statements, A print with optimizable string arguments inserted loca- tion recommended by the site-selection variable. See Figure 2 for an example.
Adding dead code. A statement having consequence its surrounding code, guarded by an i d--oodiitoonn with optimizable argument, added the program.
The last two transformations are insert transforms, which introduce new tokens in the original pro- gram, whereas the others are replace transforms, which modify existing tokens.
They implement two other transformations inserting try-catch block with optimizable parameter, and unrolling while loop once. We did not add these to final set of transforms since they are very similar to adding print statements and dead code, while producing a negligible effect. Adding every inserf transformation increases the number of variables to be optimized. Since these two transforms did not impact model performance, we omitted them to keep the number of variables to be optimized bounded.
13

---

Published as a conference paper at ICLR 2021
C AN ATTACK EXAMPLE
Unperturbed
def def_ "a, **kal: for key, value in dict("a, "*ka).items(): setattr(self, key, value) return sclf
Prediction: call
Random site-selection; Optimal site-perturbation (Ramakrishnan al., 2020)
def (sll_fl_sselff "a, "Kka]:
Prediction: call
sove, value in dict("a, "kaa).temss):: setattr(self, save, value)
return self
Optimal site-selection; Optimal site-perturbation (This work)
def_ cal__lll_eelff "copies, "*ka):
Prediction: call
for key, value in dict("copies, **ka).ttems(): setattr(self, key, value)
return self
Optimal site-selection; Optimal site-perturbation + Smoothing (This work)
def cal_callssefff 'datetime, **ka]:
Prediction: create
for key, value in dict("datetime, **ka).ttems(): setattr(self, key, value)
return self
Table 3: We present additional example which contrasts the advantages of different aspects of our formulation. In Figure [] we saw how selecting an optimal site led the optimal local variable (qisre) being found. In this example, we show how randomized smoothing, key solution we propose to ease optimization, helps in finding the best variable. In this case, just finding the optimal site is not enough to flip the classifier's is decision (call). Smoothing however enables to find local variable datet ime which flips the classifier's decision to create.
14

---

Published as a conference paper at ICLR 2021
D ADDITIONAL RESULTS
D.1 JAVA DATASET
We tabulate results of evaluating our formulation This dataset was released by Alon al. (2018) their work on CODE2SEQ. The transformations were implemented using Spoon (Pawlak et al, 2016). We find the results to be consistent with the results on the Python dataset. AO + RS provides the best attack.
additional dataset containing Java programs.
= site
h 5 sites
Method
ASR
F1
ASR
F1
No attack
0.00
100.00
0.00
100.00
Random replace
0.00
100.00
0.00
100.00
BASELINES
22.93
70.75
33.16
59.45
AO JO AO + RS 29.08
25.95
+3.02
67.17
-3.58 A
33.41
+0.25 A
59,03
-0.42 A
23.26
+0.33
70.08
-0.67 A
33.65
+0.49 A
58.85
-0.60 A
+6.15
63.90
-6.85 A
40.53
+7.37 A
51.91
-7.54 A
JO + RS
26.71
+3.78
65.41
-5.34
38.30
+5.14
53.14
-6,31 A
Table 4: Performance of our formulation dataset containing Java programs (Alon al. 2018
See Table for results on Python dataset.
D.2 FALSE POSITIVE RATE (FPR) UNDER DIFFERENT ATTACKS
We evaluated the False Positive Rate (FPR) of the model when provided perturbed programs as inputs. is important the perturbations made to the programs cause the model to start predicting ground truth tokens instead of incorrect tokens. In principle. our optimization objective strictly picks replacements which degrade the classifier's performance. As consequence, we should expect that the model does not end up perturbing the program way which leads closer to the ground truth. We empirically find the FPR consistent with this explanation. As seen in Table 55) in all our attacks, the FPR is almost O, validating that our perturbations do not introduce changes which result in the model predicting the right output.
Method attack
FPR, k=1
FPR, 4=5
0.0000
0.0000
BASELINE
0.0077
0.0130
AO 0.0073
0.0114
AO+RS
0.0075
0.0148
Table 5: False positive rates of the model under different attacks.
D.3 EFFECT OF VARIOUS TRANSFORMATIONS
We study the effect of different transformations used in our work on the attack success rate of our attacks. In our analysis, found that omitting print statements do affect the ASR of our attacks. This is helpful since for the current task we evaluate (generating program summaries), a print statement likely affects the semantics of the task.
ASR, k=1
ASR, k=5
Training
Model's Fl-score
BASELINE
AO
AO+RS
BASELINE
AO+RS
transformations (All)
33.74
19.87
23.16
30.25
37.50
43,53
$1.68
(All printt
33.14
16.39
20.22
26.46
35.14
42,70
51.24
(All vars, function params)
30.90
21.48
26.36
31.43
37.21
37.53
48.54
Table 6: The effect of variable names, function parameter names, and print statements the robustness of the learned model. These results correspond to the Python dataset.
We further investigated the effect of variable names and function parameter names on our attacks. The SEQ2SEQ model and the task of generating program summaries can appear to overly rely on the presence of variable and parameter names in the program. To empirically ascertain this, masked
15

---

Published as a conference paper at ICLR 2021
out all the variable and parameter names from our training set and trained the model with place- holder tokens. We find the model's performance to remain similar (row All vars, funct params: 1, Table (). Likewise, the attack performance also remains similar. We test another con- dition where we mask all these token names in the test set as well, during inference. We find decrease in performance (row 4, Table 6) under attack strength A 1, but the trend observed re- mains AO+RS >O BASELINE, The ASR is largely unchanged under 5.
16


Reviews:
[{"review_id": "m6c6cIlxDr8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis work tackles the problem of adversarial attacks against ML models for code-understanding tasks, such as function summarization. It formulates the problem as the adversarial application of existing semantics-preserving program transformations (e.g., renaming variables), by jointly optimizing on the location of such a transformation, and the argument to the transformation (e.g., what to replace an existing identifier with). It shows that such adversarial examples increase the attack success rate over baseline approaches, and training with such examples increases the robustness of the resulting model to the same or baseline attacks.\n\n# Strengths:\n1. Clear description of the mathematical formulation and the approach.\n1. Results over the baselines and seq2seq model seem to be very promising.\n\n# Weaknesses:\n1. The core model used is not a state-of-the-art model for code understanding. Transformer or GGNN-based models have been shown to do much better on code-understanding tasks than seq2seq models.\n1. The choice of task and perturbation actions are expedient to try out the approach, but practically questionable (see below).\n1. The choice of model on which to evaluate the approach is poor; seq2seq is a weak baseline for function-naming tasks.\n\n\n# Feedback\n\n1. As an application of techniques for adversarial robustness to code-understanding tasks, this paper is a great read. I enjoyed the description of the approach (although I'm not an expert on adversarial examples), and found the direction intuitive and well presented. I enjoyed reading your paper.\n\n1. However, digging below the surface, I'm uneasy with several aspects of the concrete realization of your approach.\n\n1. Perturbations must be semantics preserving. Although most of the perturbations you describe have that property, inserting `print` statements does not. Changing the output of a program constitutes a major modification of semantics, unless you further limit, precisely, your definition of \"semantics preserving\". As an obvious example, a function\n```\ndef ___(a):\n    result = a*2\n    return result\n```\nis very similar to \n```\ndef ___(a):\n    result = a*2\n    print(result)\n    return result\n```\nhowever I could argue that first should be named `compute_double` while the second should be named `compute_and_print_double`. I would include in this category logging statements as well; they alter the program semantics. What might not are operations that have no side effects (as far as inputs/outputs/state are concerned), for instance writing to `/dev/null`\n\n1. I'm also a little skeptical about renaming local variables and function parameters. Those may have an inordinate influence on the particular model and task you chose. However, some work has found that tokens themselves don't affect the results of some models. For example, in `Allamanis, Miltiadis, Marc Brockschmidt, and Mahmoud Khademi. \"Learning to represent programs with graphs.\" arXiv preprint arXiv:1711.00740 (2017).`, having token labels makes no difference when solving a variable-misuse task, but actually helps when solving variable naming (see Table 2).\n\n1. What's more, there's some evidence that completely removing identifiers from a program (interestingly enough, to obfuscate it) may not destroy the relevant information needed to recover the identifiers. See, for example `Bichsel, Benjamin, Veselin Raychev, Petar Tsankov, and Martin Vechev. \"Statistical deobfuscation of android applications.\" In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 343-355. 2016.` which recovers missing identifiers. In principle, that tells me that useful identifiers may still be recoverable from completely obfuscated programs from contextual structure. So this leads me to believe that what you present here is more about a model that wasn't meant to understand relational properties of its input rather focusing instead on tokens alone, and less about an inherent limitation in how we build neural models for code.\n\n1. Even ignoring the above, I'd want to see more about how imperceptibility applies to the task at hand. Your attacks may well change a predicted name from its original to something else, but how far away is that new prediction, and does it matter? When classifying, switching classes is an important effect of attack. However, when predicting a description, changing predicted tokens doesn't necessarily change the meaning of the description. What kind of change constitutes a successful attack and what kind of change doesn't disturb the correctness of the prediction? As a perhaps naive example, if you attacked a function to change its name from `read_file_from_disk`  to `collect_contents_from_storage`, although a huge change in tokens involved, I'd say the attack was unsuccessful.\n\n1. Finally, to lend credence to your results, I would want to see your approach applied to state-of-the-art models for program understanding. seq2seq is not state of the art. For variable-name prediction, something like `Alon, Uri, Shaked Brody, Omer Levy, and Eran Yahav. \"code2seq: Generating sequences from structured representations of code.\" arXiv preprint arXiv:1808.01400 (2018).` might be more appropriate.\n\n1. All in all, I learned much about adversarial training from reading your paper (since I hadn't read much about it before). However, I don't find that your results will change how models of code are built, because they are evaluated on what is now considered a fairly weak model basis, and the criteria for what constitutes a successful attack are not well defined and appear overly pessimistic.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FEfdRhjMmLZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a principled way of applying obfuscation transformation to create adversarial programs. A general mathematical formulation is presented to model the site locations and the transformation choice for each location in a perturbed program. Then, a set of first-order optimization algorithms are used to solve the proposed formulation.\n\nReasons for score:\nThe central contribution of this work is the addition of a formal approach to select perturbation locations along with the types of transformations for adversarial program generation. The mathematical formulation enables a principled approach to achieve this. However, the overall efficacy of the proposed approach over the state-of-the are demonstrated using only one model and dataset. Also, the evaluation method could use some clarification (Please see the weakness section below). I believe a more elaborate analysis will strengthen the claim and make the incremental contribution more generalizable.\n\nStrength\n1. The formulations for site-selection, site-perturbation, and perturbation strength provides a principled way of generating adversarial programs by casting it as a constrained optimization problem. It focuses on the problem of finding both optimal location and optimal transformation whereas the state-of-the-art addresses only the latter. \n2. The proposed joint optimization improves attack success rate compared to the state-of-the-art approach.\n3.  Ablation study is performed to demonstrate  the effectiveness of randomized smoothing. \n\nWeakness\n1. The comparative study would be more convincing with evaluation results from more than one model architecture and dataset.\n2. Evaluation is performed only on fully correctly classified samples. Although this approach provides a good way of understanding the perturbation effects on original positive examples, any impact on original negative examples is not visible from these experiments. I think the change in the false positive rates from the original model results could be useful to understand the impact of the generated obfuscations.\n\n\nQuestions to author: (Please address the cons above)\n1. Yefet et al. (2019) generates adversarial examples for “targeted attacks”. How effective is the proposed approach for  a “targeted attack”?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DC8xuyLjqcU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n* This paper proposes an optimization problem to adopt insert/replace operations (program obfuscations) to generate adversarial programs. They apply it to the task of program summarization, and show that they outperform the existing baseline published in 2020. In particular, one of the main contributions is the identification of the site-perturbation and site-selection process, and formalizing them as practical optimization based on PGD. \n\nStrengths:\n* The problem is relevant and challenging\n* Formalization is elegant and clear \n* Contributions and related works are relevant and significant\n* Experimental comparison against relevant, recent work \n\nWeaknesses: \n* The task of program summarization is somehow sinmple\n* It is not well-motivated why to limit the perturbation strength k, since we are talking about a program (see also discussion in [2])\n* Attack success rate improvement over baseline is moderate. \n* There seems to be no plan for the authors to publicly release the code.\n\nDetailed comments: \n* The intuition of using site-selection and site-perturbation is extremely interesting and is well developed. The formalization of the problem is very elegant, and is convincing. Modeling the “location” of the program modification and identifying that this increases attack success rate is a very important contribution.  \n* Related work is generally relevant and significant. I also strongly appreciate that there is an experimental comparison against a recent paper from 2020, representing the state of the art. As related work, I would also recommend the authors to have a look at, which seems highly related from the security domain (see [1,2,3] below). \n* I think the task of program summarization is fairly simple, and it would have been interesting to see something a bit more complex. Some “replace” operations such as renaming of variables seems to be very related to the attack in [1]. The “insertion” operation seems to be quite trivial, such as adding prints or dead code. This could be easily defeated via preprocessing commonly done by compiler’s program analysis, such as unreachable code elimination (see problem-space constraint [2]).\n* Nevertheless, I do believe that the paper proposes a very interesting perspective. I would also encourage the authors to consider releasing the source code. \n* Minor comments: \n    * Typo “reproducability” on page 6\n\nReferences:\n* [1] Quiring et al. \"Misleading authorship attribution of source code using adversarial learning.” USENIX Security Symposium, 2019.\n* [2] Pierazzi et al, “Intriguing Properties of Adversarial ML Attacks in the Problem Space”, IEEE Symp. S&P 2020\n* [3] Demetrio, Luca, et al. \"Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection.\" arXiv preprint arXiv:2008.07125 (2020).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a nice paper on generating adversarial programs. The approach is to carefully use program obfuscators. After discussion and improvements, reviewers were generally satisfied with the approach and evaluation. The problem domain was also found to be of interest.
Decision: Accept (Poster)